Method of measuring physiological parameter of subject in contactless manner

ABSTRACT

Disclosed is a method of measuring a physiological parameter in a contactless manner. The method includes acquiring a plurality of image frames for a subject, acquiring a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of image frames. The method further includes calculating a first difference and a second difference on the basis of the first color channel value, the second color channel value, and the third color channel value for at least one image frame included in the plurality of image frames. The first difference represents a difference between the first color channel value and the second color channel value for the same image frame, and the second difference represents a difference between the first color channel value and the third color channel value for the same image frame.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of U.S. PatentApplication No. 62/938,744, filed on Nov. 21, 2019, Korean PatentApplication No. 2020-0029856, filed on Mar. 10, 2020, Korean PatentApplication No. 2020-0029857, filed on Mar. 10, 2020, Korean PatentApplication No. 2020-0029858, filed on Mar. 10, 2020 and Korean PatentApplication No. 2020-0029859, filed on Mar. 10, 2020 the disclosure ofwhich is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention

Plethysmography is a technique for measuring and analyzing naturalchanges in shape or form when the volume of human tissues such as humanorgans or blood vessels changes according to the flow of blood vessels.

2. Discussion of Related Art

The most common technique for measuring photoplethysmography (PPG) usinglight uses a method of analyzing the amount of transmitted light withrespect to the amount of light emitted to a human body, and this isexplained by the Beer-Lambert law in which light absorbance isproportional to the concentration of absorbing material and thethickness of an absorbing layer. According to this law, the change intransmitted light results in a signal proportional to the change in thevolume of light-transmitting material, and thus it is possible to checka state of a human heart by using PPG even when the absorbance of thematerial is not known.

Recently, a technique using remote photoplethysmography (rPPG), which isone step evolved from the technique using PPG, has emerged. As the mostpopular technique to check a signal related to a heartbeat using PPG;there is a technique for acquiring PPG by bringing a device having acamera and a light that are close to each other and attached thereto,such as a smartphone, into direct contact with a human body, emittinglight, and measuring transmitted light. Recently, a technology relatedto remote photoplethysmography (rPPG) to check a change in the volume ofa blood vessel in a signal acquired from an image captured by a camerais continuously being researched and developed.

The technique using rPPG can be variously applied to devices and placesequipped with cameras, such as airport immigration offices and remotemedical treatments, in that no contact between a subject and ameasurement instrument is required.

However, in the technique related to rPPG noise that is caused byambient light and subject movement while a subject is captured with acamera has a large effect on a signal, and thus a technique forextracting only a signal related to the change in volume of a subject tobe measured from the captured image can be regarded as a core techniquefor a technique for measuring a physiological signal using rPPG.

SUMMARY OF THE INVENTION

A problem to be solved according to an embodiment is to acquire aphysiological parameter in a contactless manner.

A problem to be solved according to another embodiment is to reducenoise caused by subject movement in order to acquire a physiologicalparameter.

A problem to be solved according to still another embodiment is toreduce noise caused by a change in intensity of external light in orderto acquire a physiological parameter.

A problem to be solved according to still another embodiment is toacquire various physiological parameters at the same time.

A problem to be solved according to still another embodiment is toacquire physiological information based on various physiologicalparameters.

A problem to be solved according to still another embodiment is toacquire various physiological parameters in association with each otherat the same time.

A problem to be solved according to still another embodiment is todetect drowsiness on the basis of an LF/HF ratio of a heartbeat signaland a heart rate of a subject.

A problem to be solved according to still another embodiment is relatedto a smart mirror device for acquiring at least two associatedphysiological parameters.

A problem to be solved according to still another embodiment is relatedto a smart mirror device operating method to acquire at least twoassociated physiological parameters.

A problem to be solved according to still another embodiment is relatedto a smart mirror device including a switch device.

According to an aspect of the present invention, there is provided amethod of measuring a physiological parameter in a contactless manner,the method including acquiring a plurality of image frames for asubject, acquiring a first color channel value, a second color channelvalue, and a third color channel value for at least one image frameincluded in the plurality of image frames, calculating a firstdifference and a second difference on the basis of the first colorchannel value, the second color channel value, and the third colorchannel value for at least one image frame included in the plurality ofimage frames, wherein the first difference represents a differencebetween the first color channel value and the second color channel valuefor the same image frame, and the second difference represents adifference between the first color channel value and the third colorchannel value for the same image frame, acquiring a first characteristicvalue on the basis of the first difference for at least one image frameincluded in a first image frame group acquired during a first presettime period and the mean of first differences for the first image framegroup, acquiring a second characteristic value on the basis of thesecond difference for at least one image frame included in the firstimage frame group and the mean of second differences for the first imageframe group, and determining a physiological parameter of the subject onthe basis of the first characteristic value and the secondcharacteristic value, wherein the first color channel value mayrepresent an average pixel value of a first color channel for one imageframe, the second color channel value may represent an average pixelvalue of a second color channel for one image frame, and the third colorchannel value may represent an average pixel value of a third colorchannel for one image frame.

According to an aspect of the present invention, there is provided amethod of measuring a physiological parameter using an infrared camera,the method including acquiring a plurality of image frames for a subjectusing an infrared camera, acquiring a first region value, a secondregion value, and a third region value for at least one image frameincluded in the plurality of image frames, calculating a firstdifference and a second difference on the basis of the first regionvalue, the second region value, and the third region value for at leastone image frame included in the plurality of image frames, acquiring afirst characteristic value on the basis of the first difference for atleast one image frame included in a first image frame group acquiredduring a first preset time period and the mean of first differences forthe first image frame group, acquiring a second characteristic value onthe basis of the second difference for at least one image frame includedin the first image frame group and the mean of second differences forthe first image frame group, and determining a physiological parameterof the subject on the basis of the first characteristic value and thesecond characteristic value, wherein the first region value may be anaverage pixel value of a first region of interest for one image frame,the second region value may be an average pixel value of a second regionof interest for one image frame, the third region value may be anaverage pixel value of a third region of interest for one image frame,the first difference may be a difference between the first region valueand the second region value for the same image frame, and the seconddifference may be a difference between the first region value and thethird region value for the same image frame.

According to an aspect of the present invention, there is provided aphysiological parameter acquisition device including an imageacquisition unit for acquiring an image frame for a subject and acontrol unit for acquiring a physiological parameter using the imageframe, wherein the control unit is configured to acquire a first colorchannel value, a second color channel value, and a third color channelvalue for at least one image frame included in a plurality of acquiredimage frames, calculate a first difference and a second difference onthe basis of the first color channel value, the second color channelvalue, and the third color channel value for at least one image frameincluded in the plurality of image frames, acquire a firstcharacteristic value on the basis of the first difference for at leastone image frame included in a first image frame group acquired during afirst preset time period and the mean of first differences for the firstimage frame group, acquire a second characteristic value on the basis ofthe second difference for at least one image frame included in the firstimage frame group and the mean of second differences for the first imageframe group, and determine a physiological parameter of the subject onthe basis of the first characteristic value and the secondcharacteristic value, wherein the first color channel value may be anaverage pixel value of a first color channel for one image frame, thesecond color channel value may be an average pixel value of a secondcolor channel for one image frame, the third color channel value may bean average pixel value of a third color channel for one image frame, thefirst difference may be a difference between the first color channelvalue and the second color channel value for the same image frame, andthe second difference may be a difference between the first colorchannel value and the third color channel value for the same imageframe.

According to an aspect of the present invention, there is provided amethod of providing a physiological parameter in a contactless manner,the method including acquiring a plurality of image frames for asubject, acquiring a first color channel value, a second color channelvalue, and a third color channel value for at least one image frameincluded in the plurality of image frames, acquiring a firstcharacteristic value on the basis of a first color channel value, asecond color channel value, and a third color channel value for at leastone image frame included in a first image frame group acquired during afirst preset time period, acquiring a second characteristic value on thebasis of a first color channel value, a second color channel value, anda third color channel value for at least one image frame included in asecond image frame group acquired during a second preset time period,and determining a physiological parameter on the basis of the firstcharacteristic value and the second characteristic value, wherein thefirst image frame group and the second image frame group may partiallyoverlap each other, and a first characteristic value for a first imageframe included in the first image frame group but not included in thesecond image frame group, a first characteristic value and a secondcharacteristic value for a second image frame included in both of thefirst image frame group and the second image frame group, and a secondcharacteristic value for a third image frame included in the secondimage frame group but not included in the first image frame group may beused in order to determine the physiological parameter.

According to an aspect of the present invention, there is provided amethod of measuring a physiological parameter in a contactless manner,the method including acquiring a plurality of image frames including afirst image frame group and a second image frame group at leastpartially overlapping the first image frame group, acquiring aphysiological parameter on the basis of the first image frame group,outputting the physiological parameter at a first time point, acquiringa first physiological parameter on the basis of the second image framegroup, and outputting the physiological parameter at a second time pointlater than the first time point, wherein the physiological parameteroutput at the first time point may be a physiological parameter acquiredbased on the first image frame group, the physiological parameter outputat the second time point may be the first physiological parameter when adifference between the first physiological parameter and thephysiological parameter output at the first time point is less than orequal to a reference value and may be a physiological parameter obtainedby correcting the physiological parameter output at the first time pointwhen the difference between the first physiological parameter and thephysiological parameter output at the first time point is greater thanthe reference value.

According to an aspect of the present invention, there is provided amethod of measuring a physiological parameter, the method includingacquiring a plurality of image frames for a subject, setting a firstregion and a second region for at least one image frame included in theplurality of image frames, determining an oxygen saturation level of thesubject on the basis of a first feature acquired based on at least twoof a first color channel value, a second color channel value, and athird color channel value for the first region, determining a heart rateof the subject on the basis of a second feature acquired based on afirst difference, which is a difference between the first color channelvalue and the second color channel value for the first region,determining a blood pressure of the subject on the basis of a thirdfeature acquired based on the first difference and a second difference,which is a difference between a first color channel value and a secondcolor channel value for the second region, and outputting the oxygensaturation level, the heart rate, and the blood pressure, wherein thefirst feature may be acquired based on a first image frame groupacquired during a first time period, the second feature may be acquiredbased on a second image frame group acquired during a second timeperiod, the third feature may be acquired based on a third image framegroup acquired during a third time period, and the first image framegroup, the second image frame group, and the third image frame group mayinclude a plurality of image frames in common to acquire the oxygensaturation level, the heart rate, and the blood pressure in associationwith each other.

According to another aspect of the present invention, there is provideda method of measuring a physiological parameter, the method includingacquiring a plurality of image frames for a subject, setting a firstregion, a second region, and a third region for at least one image frameincluded in the plurality of image frames, determining an oxygensaturation level of the subject on the basis of a first feature acquiredbased on at least two of a first color channel value, a second colorchannel value, and a third color channel value for the first region,determining a heart rate of the subject on the basis of a second featureacquired based on a first difference, which is a difference between thefirst color channel value and the second color channel value for thefirst region, determining a blood pressure of the subject on the basisof a third feature acquired based on a second difference, which is adifference between a first color channel value and a second colorchannel value for the second region, and a third difference, which is adifference between a first color channel value and a second colorchannel value for the third region, and outputting the oxygen saturationlevel, the heart rate, and the blood pressure, wherein the first featuremay be acquired based on a first image frame group acquired during afirst time period, the second feature may be acquired based on a secondimage frame group during a second time period, the third feature may beacquired based on a third image frame group during a third time period,and the first image frame group, the second image frame group, and thethird image frame group may include a plurality of image frames incommon to acquire the oxygen saturation level, the heart rate, and theblood pressure in association with each other.

According to still another aspect of the present invention, there isprovided a method of measuring a physiological parameter, the methodincluding acquiring a plurality of image frames for a subject, acquiringa first color channel value, a second color channel value, and a thirdcolor channel value for at least one image frame included in theplurality of image frames, determining an oxygen saturation level of thesubject on the basis of a first feature acquired based on at least twoof the first color channel value, the second color channel value, andthe third color channel value, determining a heart rate of the subjecton the basis of a second feature acquired based on a first difference,which is a difference between the first color channel value and thesecond color channel value, and a second difference, which is adifference between the first color channel value and the third colorchannel value, determining a blood pressure of the subject on the basisof a third feature acquired based on the first difference and the seconddifference, and outputting the oxygen saturation level, the heart rate,and the blood pressure, wherein the first feature may be acquired basedon a first image frame group acquired during a first time period, thesecond feature may be acquired based on a second image frame groupacquired during a second time period, the third feature may be acquiredbased on a third image frame group acquired during a third time period,and the first image frame group, the second image frame group, and thethird image frame group may include a plurality of image frames incommon to acquire the oxygen saturation level, the heart rate, and theblood pressure in association with each other.

According to still another aspect of the present invention, there isprovided a method of measuring a physiological parameter, the methodincluding acquiring a plurality of image frames for a subject, settingat least two regions for at least one image frame included in theplurality of image frames, acquiring a first color channel value, asecond color channel value, a third color channel value, a fourth colorchannel value, and a fifth color channel value for at least one imageframe included in the plurality of image frames, determining an oxygensaturation level of the subject on the basis of a first feature acquiredbased on at least two of the first color channel value, the second colorchannel value, and the third color channel value, determining a heartrate of the subject on the basis of a second feature acquired based onat least two of the first color channel value, the second color channelvalue, and the third color channel value, determining a blood pressureof the subject on the basis of a third feature acquired based on atleast two of the first color channel value, the second color channelvalue, and the third color channel value, determining a core temperatureof the subject on the basis of a fourth feature acquired based on thefourth color channel value, and outputting the oxygen saturation level,the heart rate, the core temperature, and the blood pressure, whereinthe first color channel value may be a green channel value, the secondchannel value may be a red channel value, the third color channel valuemay be a blue channel value, the fourth color channel value may be asaturation channel value, and the fifth color channel value may be a huechannel value.

According to still another aspect of the present invention, there isprovided a method of measuring a physiological parameter, the methodincluding acquiring a plurality of image frames for a subject, acquiringN preliminary heart rates on the basis of at least one image frameincluded in the plurality of image frames, acquiring M preliminaryoxygen saturation levels on the basis of at least one image frameincluded in the plurality of image frames, acquiring K preliminary bloodpressures on the basis of at least one image frame included in theplurality of image frames, acquiring a heart rate on the basis of the Npreliminary heart rates, acquiring an oxygen saturation level on thebasis of the M preliminary oxygen saturation levels, acquiring a bloodpressure on the basis of the K preliminary blood pressures, andoutputting the heart rate, the oxygen saturation level, and the bloodpressure, wherein when an image frame acquired when the subject is in afirst state is a first image frame, the first image frame may beincluded in common in the image frames used to obtain the N preliminaryheart rates, the NI preliminary oxygen saturation levels, and the Kblood pressures.

According to an aspect of the present invention, there is provided amethod of acquiring physiological information, the method includingacquiring a plurality of image frames for a subject, acquiring a firstphysiological parameter on the basis of a first image frame groupincluding at least one image frame included in the plurality of imageframes, acquiring a second physiological parameter on the basis of asecond image frame group including at least one image frame included inthe plurality of image frames, acquiring physiological information onthe basis of at least one of the first physiological parameter and thesecond physiological parameter, and outputting the first physiologicalparameter, the second physiological parameter, and the physiologicalinformation, wherein the first image frame group and the second imageframe group may at least partially overlap each other in order toacquire physiological information in response to a specific state of thesubject.

According to an aspect of the present invention, there is provided amethod of detecting drowsiness based on a heart rate, the method beingperformed by at least one processor and including acquiring a heart rateof a subject, acquiring a comparison result obtained by comparing theheart rate to a reference heart rate, acquiring a duration for which theheart rate is less than or equal to the reference heart rate on thebasis of the comparison result, and a drowsiness detection operation fordetermining a drowsiness state of the subject on the basis of whetherthe duration reaches a reference duration.

According to an aspect of the present invention, there is provided amethod of detecting drowsiness based on a heart rate and a low frequency(LF)/high frequency (HF) ratio, the method being performed by at leastone processor and including acquiring a heart rate of a subject,acquiring a comparison result obtained by comparing the heart rate to areference heart, rate, acquiring a first drowsiness parameter on thebasis of the comparison result, acquiring an LF/HF ratio representing aratio of a sympathetic nerve activity and a parasympathetic nerveactivity of the subject, acquiring a second drowsiness parameter on thebasis of the LF/HF ratio of the subject, and a drowsiness detectionoperation for determining a drowsiness state of the subject using atleast one of the first drowsiness parameter and the second drowsinessparameter.

According to an aspect of the present invention, there is provided asmart mirror device including a reflective mirror surface, an imageacquisition unit for acquiring a plurality of image frames for asubject, a display unit placed behind the reflective mirror surface andconfigured to display visual information through the reflective mirrorsurface, and a control unit configured to control the operation of theimage acquisition unit and the display unit and acquire a physiologicalparameter in a contactless manner, wherein the control unit may controlthe display unit so that a first physiological parameter acquired basedon a first image frame group included in the plurality of image framesat a first time point is displayed and control the display unit so thata second physiological parameter acquired based on a second image framegroup included in the plurality of image frames at a second time pointis displayed, the first image frame group and the second image framegroup may include at least one image frame in common to associate thefirst physiological parameter and the second physiological parameterwith each other, and the at least one image frame included in the firstimage frame group and the second image frame group in common may includean image frame acquired in a first state of the subject observed by thesubject through the reflective mirror surface before the first timepoint and the second time point.

According to an aspect of the present invention, there is provided amethod of operating a smart mirror device, the method includingacquiring an on-trigger, acquiring a plurality of image frames for asubject, acquiring a first physiological parameter on the basis of afirst image frame group included in the plurality of image frames,acquiring a second physiological parameter on the basis of a secondimage frame group included in the plurality of image frames, displayingthe first physiological parameter and the second physiologicalparameter, acquiring an off-trigger, and stopping the acquisition of theplurality of image frames for the subject, wherein the first image framegroup and the second image frame group may include at least one imageframe in common to associate the first physiological parameter and thesecond physiological parameter with each other, the on-trigger may beacquired from at least one sensor, and the off-trigger may be acquiredfrom an image sensor for acquiring the plurality of image frames.

According to another aspect of the present invention, there is provideda smart mirror device including a reflective mirror surface, an imageacquisition unit for acquiring a plurality of image frames, a displayunit placed behind the reflective mirror surface and configured todisplay visual information through the reflective mirror surface, aswitch unit placed in front of the image acquisition unit and configuredto switch a field of view of the image acquisition unit, and a controlunit configured to control operations of the image acquisition unit andthe display unit and acquire a physiological parameter, wherein theswitch unit may have a surface formed as a reflective mirror, and whenthe switch unit is open, the control unit may control the display unitso that the physiological parameter and at least one piece of visualinformation are displayed through the display unit, and when the switchunit is closed, the control unit may control the display unit so that atleast one piece of visual information is displayed through the displayunit.

Solutions of the present invention are not limited to theabove-mentioned solutions, and solutions that have not been mentionedwill be clearly understood by those skilled in the art from thefollowing description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings, in which:

FIG. 1 is a diagram showing a physiological-parameter andphysiological-information management system according to an embodiment;

FIG. 2 is a diagram showing a physiological-parameter andphysiological-information management system according to anotherembodiment;

FIG. 3 is a diagram illustrating a physiological parameter acquisitiondevice according to an embodiment;

FIG. 4 is a flowchart showing a physiological parameter acquisitionmethod according to an embodiment;

FIG. 5 is a diagram showing a physiological parameter acquisition methodaccording to an embodiment;

FIG. 6 is a flowchart showing a physiological information acquisitionmethod according to an embodiment;

FIGS. 7 and 8 are diagrams illustrating a physiological parameteracquisition method using a physiological parameter acquisition model;

FIG. 9 is a flowchart illustrating a heart rate measurement methodaccording to an embodiment;

FIG. 10 is a flowchart illustrating an oxygen saturation levelmeasurement method according to an embodiment;

FIG. 11 is a flowchart illustrating an oxygen saturation levelmeasurement method according to another embodiment;

FIG. 12 is a flowchart illustrating a blood pressure measurement methodaccording to an embodiment;

FIG. 13 is a flowchart illustrating a blood pressure measurement methodaccording to another embodiment;

FIG. 14 is a flowchart strafing a core temperature measurement methodaccording to an embodiment;

FIG. 15 is a flowchart illustrating a heart rate acquisition methodaccording to an embodiment;

FIG. 16 is a graph of color channel values according to an embodiment;

FIG. 17 is a graph showing a noise reduction method according to anembodiment;

FIG. 18 is a diagram showing the absorbance of hemoglobin andoxyhemoglobin in a visible light range;

FIG. 19 is a diagram illustrating a characteristic value acquisitionmethod according to an embodiment;

FIG. 20 is a diagram illustrating a characteristic value acquisitionmethod according to another embodiment;

FIG. 21 is a diagram illustrating a method of using a plurality ofcharacteristic values;

FIG. 22 is a graph showing a frequency component extracted from a graphfor a characteristic value;

FIG. 23 is a diagram illustrating a heart rate acquisition methodaccording to an embodiment;

FIG. 24 is a flowchart illustrating an output heart rate correctionmethod according to an embodiment;

FIG. 25 is a diagram illustrating a heartbeat signal extraction methodaccording to an embodiment;

FIG. 26 is a diagram illustrating a heart rate acquisition method usinginfrared light according to an embodiment;

FIG. 27 is a diagram illustrating a heart rate acquisition method usinginfrared light according to an embodiment;

FIG. 28 is a flowchart illustrating a physiological parameteracquisition method according to an embodiment;

FIG. 29 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters and a plurality of pieces of physiologicalinformation according to an embodiment;

FIG. 30 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 31 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 32 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 33 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 34 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 35 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 36 is a diagram illustrating a method of acquiring a plurality ofassociated physiological parameters according to an embodiment;

FIG. 37 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment;

FIG. 38 is a block diagram of a drowsiness detection device on the basisof a heart rate;

FIG. 39 is a flowchart of a method of detecting drowsiness on the basisof a heart rate;

FIG. 40 is a graph of an average heart rate of a subject on the basis ofa heart rate of the subject of the measurement;

FIG. 41 is a graph of a heart rate of a subject for explaining asituation in which the subject is detected to be in a drowsiness stateon the basis of a heart rate.

FIG. 42 is a graph of a heart rate including noise;

FIG. 43 is a graph illustrating a situation in which a subject hasrecovered to a normal state;

FIG. 44 is a flowchart of a method of detecting drowsiness on the basisof a low frequency (LF)/high frequency (HF) ratio;

FIG. 45 is a graph of an LF/HF ratio of a subject for explaining asituation in which the subject is detected to be in a drowsiness stateon the basis of an LF/HF ratio;

FIG. 46 is a graph of an LF/HF ratio of a subject in order to representa situation in which the subject has recovered from a drowsiness stateon the basis of an LF/HF ratio;

FIG. 47 is a flowchart of a method of detecting drowsiness on the basisof a heart rate and an LF/HF ratio;

FIG. 48 is a diagram illustrating a smart mirror device according to anembodiment;

FIG. 49 is a diagram illustrating a smart mirror device according to anembodiment;

FIG. 50 is a diagram illustrating a smart mirror device in which a guideregion is output according to an embodiment;

FIG. 51 is a diagram illustrating a smart mirror device in whichpredetermined information is output according to an embodiment;

FIG. 52 is a diagram illustrating a smart mirror device according to anembodiment;

FIG. 53 is a diagram illustrating a display device configured to measurea physiological parameter in real time according to an embodiment;

FIG. 54 is a diagram illustrating a smart mirror device in whichpredetermined information is output according to an embodiment;

FIG. 55 is a diagram illustrating a smart mirror device including aswitching device according to an embodiment;

FIG. 56 is a diagram illustrating a smart mirror device placed above ashoe rack according to an embodiment;

FIG. 57 is a flowchart illustrating a smart mirror device operatingmethod according to an embodiment;

FIGS. 58 and 59 are diagrams illustrating an operation of a smart mirrordevice using a trigger signal according to an embodiment;

FIG. 60 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment;

FIG. 61 is a diagram illustrating an operation of a smart mirror deviceaccording to an embodiment;

FIG. 62 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment;

FIG. 63 is a diagram illustrating an operation of a smart mirror deviceaccording to an embodiment;

FIG. 64 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment;

FIGS. 65 and 66 are diagrams illustrating an operation of a smart mirrordevice according to an embodiment;

FIG. 67 is a diagram illustrating a physiological parameter measurementdevice according to an embodiment;

FIG. 68 is a diagram illustrating a physiological parameter measurementdevice according to an embodiment;

FIG. 69 is a diagram illustrating a physiological parameter measurementdevice placed on an autonomous vehicle according to an embodiment;

FIG. 70 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 71 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 72 is a diagram illustrating a driving scheduling assistance deviceoperating method using a physiological parameter measurement deviceaccording to an embodiment;

FIG. 73 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 74 is a flowchart illustrating a driving parameter calculationdevice operating method according to an embodiment;

FIG. 75 is a diagram illustrating an infant monitoring device accordingto an embodiment;

FIG. 76 is a diagram illustrating the occurrence of an event in relationto an infant;

FIG. 77 is a flowchart illustrating an infant monitoring deviceoperating method according to an embodiment;

FIG. 78 is a flowchart illustrating an infant monitoring deviceoperating method according to an embodiment;

FIG. 79 is a diagram showing a mobile application for implementing aninfant monitoring system according to an embodiment;

FIG. 80 is a diagram illustrating a physiological parameter measurementdevice placed in a reading room according to an embodiment;

FIG. 81 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 82 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 83 is a diagram illustrating a physiological parameter measurementdevice used for cognitive rehabilitation therapy according to anembodiment;

FIG. 84 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 85 is a diagram illustrating a physiological parameter measurementdevice used for immigration screening according to an embodiment;

FIG. 86 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment;

FIG. 87 is a diagram illustrating a physiological parameter measurementdevice used for a security device according to an embodiment;

FIG. 88 is a flowchart illustrating a security device operating methodaccording to an embodiment;

FIG. 89 is a diagram illustrating a physiological parameter measurementdevice used for a kiosk according to an embodiment; and

FIG. 90 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Embodiments described in this specification are intended to clearlyexplain the spirit of the invention to those skilled in the art.Therefore, the present invention is not limited by the embodiments, andthe scope of the present invention should be interpreted as encompassingmodifications and variations without departing from the spirit of theinvention.

Terms used in this specification are selected from among general terms,which are currently widely used, in consideration of functions in thepresent invention and may have meanings varying depending on intentionsof those skilled in the art, customs in the field of art, the emergenceof new technologies, or the like. If a specific term is used with aspecific meaning, the meaning of the term will be describedspecifically. Accordingly, the terms used in this specification shouldnot be defined as simple names of the components but should be definedon the basis of the actual meaning of the terms and the whole contextthroughout the present specification.

The accompanying drawings are for facilitating the explanation of thepresent invention, and the shape in the drawings may be exaggerated forthe purpose of convenience of explanation, so the present inventionshould not be limited by the drawings.

When it is determined that detailed descriptions of well-known elementsor functions related to the present invention may obscure the subjectmatter of the present invention, detailed descriptions thereof will beomitted herein as necessary.

According to an embodiment, there may be provided a method of measuringa physiological parameter in a contactless manner, the method includingacquiring a plurality of image frames for a subject, acquiring a firstcolor channel value, a second color channel value, and a third colorchannel value for at least one image frame included in the plurality ofimage frames, calculating a first difference and a second difference onthe basis of the first color channel value, the second color channelvalue, and the third color channel value for at least one image frameincluded in the plurality of image frames, wherein the first differencerepresents a difference between the first color channel value and thesecond color channel value for the same image frame, and the seconddifference represents a difference between the first color channel valueand the third color channel value for the same image frame, acquiring afirst characteristic value on the basis of the first difference for atleast one image frame included in a first image frame group acquiredduring a first preset time period and. the mean of first differences forthe first image frame group, acquiring a second characteristic value onthe basis of the second difference for at least one image frame includedin the first image frame group and the mean of second differences forthe first image frame group, and determining a physiological parameterof the subject on the basis of the first characteristic value and thesecond characteristic value, wherein the first color channel value mayrepresent an average pixel value of a first color channel for one imageframe, the second color channel value may represent an average pixelvalue of a second color channel for one image frame, and the third colorchannel value may represent an average pixel value of a third colorchannel for one image frame.

Here, the physiological parameter may include at least one of a heartrate and a blood. pressure.

Here, the first color channel, the second color channel, and the thirdcolor channel may be color channels of an RGB color space.

Here, the first color channel may be set to a green channel, the secondcolor channel may be set to a red channel, and the third color channelmay be set to a blue channel in order to reduce noise in considerationof the absorbance of hemoglobin and oxyhemoglobin.

Here, the first characteristic value may be acquired based on a firstdeviation of the first difference for at least one image frame includedin the first image frame group, the second characteristic value may beacquired based on a second deviation of the second difference for atleast one image frame included the first image frame group, the firstdeviation may be calculated based on the first difference for the atleast one image frame and the mean of first differences for the firstimage frame group, and the second deviation may be calculated based onthe second difference for the at least one image frame and the mean ofsecond differences for the first image frame group.

Here, the first characteristic value and the second characteristic valuemay be normalized values.

Here, the first characteristic value may be a value normalized by afirst standard deviation, the second characteristic value may be a valuenormalized by a second standard deviation, the first standard deviationmay be the standard deviation of the first difference for the firstimage frame group, and the second standard deviation may be the standarddeviation of the second difference for the first image frame group.

Here, the physiological parameter of the subject may be determined basedon a third characteristic value acquired by summing the firstcharacteristic value and the second characteristic value.

Here, the method may further include outputting the physiologicalparameter of the subject, the determined physiological parameter mayinclude a first physiological parameter and a second physiologicalparameter, and the output physiological parameter may be determinedbased on the first physiological parameter and the second physiologicalparameter.

Here, the first physiological parameter may be determined based on asecond image frame group, the second physiological parameter may bedetermined based on a third image frame group, the number of imageframes included in the first image frame group may be smaller than thenumber of image frames included in the second image frame group and thethird image frame group, and the first image frame group may be includedin the second image frame group.

Here, the number of image frames included in the second image framegroup may be equal to the number of image frames included in the thirdimage frame group.

Here, the method may further include outputting the physiologicalparameter based on the determined physiological parameter, thedetermined physiological parameter may include at least four preliminaryphysiological parameters, and the output physiological parameter may bedetermined. based on the four preliminary physiological parameters.

Here, the method may further include outputting the physiologicalparameter based on the determined physiological parameter, the outputphysiological parameter may include a first physiological parameter anda second physiological parameter, the second physiological parameter maybe a physiological parameter of the same type as the first physiologicalparameter, the second physiological parameter may be output after thefirst physiological parameter is output, and when a difference betweenthe second physiological parameter and the first physiological parameterexceeds a reference value, the second physiological parameter may becorrected and output.

According to another embodiment, there is provided a method of measuringa physiological parameter in a contactless manner using an infraredcamera, the method including acquiring a plurality of image frames for asubject using an infrared camera, acquiring a first region value, asecond region value, and a third region value for at least one imageframe included in the plurality of image frames, calculating a firstdifference and a second difference on the basis of the first regionvalue, the second region value, and the third region value for at leastone image frame included in the plurality of image frames, acquiring afirst characteristic value on the basis of the first difference for atleast one image frame included in a first image frame group acquiredduring a first preset time period and the mean of first differences forthe first image frame group, acquiring a second characteristic value onthe basis of the second difference for at least one image frame includedin the first image frame group and the mean of second differences forthe first image frame group, and determining a physiological parameterof the subject on the basis of the first characteristic value and thesecond characteristic value, wherein the first region value may be anaverage pixel value of a first region of interest for one image frame,the second region value may be an average pixel value of a second regionof interest for one image frame, the third region value may be anaverage pixel value of a third region of interest for one image frame,the first difference may be a difference between the first region valueand the second region value for the same image frame, and the seconddifference may be a difference between the first region value and thethird region value for the same image frame.

Here, the physiological parameter may include at least one of a heartrate and a blood pressure.

Here, the first characteristic value may be acquired based on a firstdeviation of the first difference for at least one image frame includedin the first image frame group, the second characteristic value may beacquired based on a second deviation of the second difference for atleast one image frame included the first image frame group, the firstdeviation may be calculated based on the first difference for the atleast one image frame and the mean of first differences for the firstimage frame group, and the second deviation may be calculated based onthe second difference for the at least one image frame and the mean ofsecond differences for the first image frame group.

According to still another embodiment, there may be provided aphysiological parameter measurement device for measuring a physiologicalparameter in a contactless manner, the physiological parametermeasurement device including an image acquisition unit for acquiring animage frame for a subject and a control unit for acquiring aphysiological parameter using the image frame, wherein the control unitis configured to acquire a first color channel value, a second colorchannel value, and a third color channel value for at least one imageframe included in a plurality of acquired image frames, calculate afirst difference and a second difference on the basis of the first colorchannel value, the second color channel value, and the third colorchannel value for at least one image frame included in the plurality ofimage frames, acquire a first characteristic value on the basis of thefirst difference for at least one image frame included in a first imageframe group acquired during a first preset time period and the mean offirst differences for the first image frame group, acquire a secondcharacteristic value on the basis of a second difference for at leastone image frame included in the first image frame group and the mean ofsecond differences for the first image frame group, and determine aphysiological parameter of the subject on the basis of the firstcharacteristic value and the second characteristic value, wherein thefirst color channel value may be an average pixel value of a first colorchannel for one image frame, the second color channel value may be anaverage pixel value of a second color channel for one image frame, thethird color channel value may be an average pixel value of a third colorchannel for one image frame, the first difference may be a differencebetween the first color channel value and the second color channel valuefor the same image frame, and the second difference may be a differencebetween the first color channel value and the third color channel valuefor the same image frame.

Here, the first color channel, the second color channel, and the thirdcolor channel may be color channels of an RGB color space, and the firstcolor channel may be set to a green channel, the second color channelmay be set to a red channel, and the third color channel may be set to ablue channel in order to reduce noise in consideration of the absorbanceof hemoglobin and oxyhemoglobin.

Here, the first characteristic value may be acquired based on a firstdeviation of the first difference for at least one image frame includedin the first image frame group, the second characteristic value may beacquired based on a second deviation of the second difference for atleast one image frame included the first image frame group, the firstdeviation may be calculated based on the first difference for the atleast one image frame and the mean of first differences for the firstimage frame group, and the second deviation may be calculated based onthe second difference for the at least one image frame and the mean ofsecond differences for the first image frame group.

Here, the control unit may acquire physiological information on thebasis of the determined physiological parameter, and the physiologicalinformation may include at least one of emotion information, drowsinessinformation, stress information, and excitement information.

According to still another embodiment, there may be provided a method ofmeasuring a physiological parameter in a contactless manner, the methodincluding acquiring a plurality of image frames for a subject, acquiringa first color channel value, a second color channel value, and a thirdcolor channel value for at least one image frame included in theplurality of image frames, acquiring a first characteristic value on thebasis of a first color channel value, a second color channel value, anda third color channel value for at least one image frame included in afirst image frame group acquired during a first preset time period,acquiring a second characteristic value on the basis of a first colorchannel value, a second color channel value, and a third color channelvalue for at least one image frame included in a second image framegroup acquired during a second preset time period, and determining aphysiological parameter on the basis of the first characteristic valueand the second characteristic value, wherein the first image frame groupand the second image frame group may partially overlap each other, and afirst characteristic value for a first image frame included in the firstimage frame group but not included in the second image frame group, afirst characteristic value and a second characteristic value for asecond image frame included in both of the first image frame group andthe second image frame group, and a second characteristic value for athird image frame included in the second image frame group but notincluded in the first image frame group may be used in order todetermine the physiological parameter.

Here, the method may further include calculating a first difference onthe basis of at least some of the first color channel value, the secondcolor channel value, and the third color channel value for the at leastone image frame included in the plurality of image frames, wherein thefirst characteristic value may be acquired based on the first differencefor the at least one image frame included in the first image frame groupand the mean of first differences for the first image frame group, thesecond characteristic value may be acquired based on the firstdifference for the at least one image frame included in the second imageframe group and the mean of second differences for the second imageframe group, the first difference may be a difference between a firstcolor channel value and a second color channel value for the same imageframe, and the second difference may be a difference between a firstcolor channel value and a third color channel value for the same imageframe.

Here, the first characteristic value may be acquired based on a firstdeviation of the first difference for at least one image frame includedin the first image frame group, the second characteristic value may beacquired based on a second deviation of the first difference for atleast one image frame included the second image frame group, the firstdeviation may be calculated based on the first difference for the atleast one image frame included in the first image frame group and themean of first differences for the first image frame group, and thesecond deviation may be calculated based on the first difference for theat least one image frame included in the second image frame group andthe mean of first differences for the second image frame group.

According to still another embodiment, there may be provided aphysiological parameter output method for acquiring and outputting aphysiological parameter in a contactless manner, the physiologicalparameter output method including acquiring a plurality of image framesincluding a first image frame group and a second image frame group atleast partially overlapping the first image frame group, acquiring aphysiological parameter on the basis of the first image frame group,outputting the physiological parameter at a first time point, acquiringa first physiological parameter on the basis of the second image framegroup, and outputting the physiological parameter at a second time pointlater than the first time point, wherein the physiological parameteroutput at the first time point may be a physiological parameter acquiredbased on the first image frame group, the physiological parameter outputat the second time point may be the first physiological parameter when adifference between the first physiological parameter and thephysiological parameter output at the first time point is less than orequal to a reference value and may be a physiological parameter obtainedby correcting the physiological parameter output at the first time pointwhen the difference between the first physiological parameter and thephysiological parameter output at the first time point is greater thanthe reference value.

Here, the corrected physiological parameter may be a physiologicalparameter corrected by adding a preset value to the physiologicalparameter output at the first time point when the first physiologicalparameter is greater than the physiological parameter output at thefirst time point and may be a physiological parameter corrected bysubtracting a preset value from the physiological parameter output atthe first time when the first physiological parameter is smaller thanthe physiological parameter output at the first time point.

According to an embodiment, there may be provided a method of measuringvarious physiological parameters including a heart rate, an oxygensaturation level, and a blood pressure at the same time, the methodincluding acquiring a plurality of image frames for a subject, setting afirst region and a second region for at least one image frame includedin the plurality of image frames, determining an oxygen saturation levelof the subject on the basis of a first feature acquired based on atleast two of a first color channel value, a second color channel value,and a third color channel value for the first region, determining aheart rate of the subject on the basis of a second feature acquiredbased on a first difference, which is a difference between the firstcolor channel value and the second color channel value for the firstregion, determining a blood pressure of the subject on the basis of athird feature acquired based on the first difference and a seconddifference, which is a difference between a first color channel valueand a second color channel value for the second region, and outputtingthe oxygen saturation level, the heart rate, and the blood pressure,wherein the first feature may be acquired based on a first image framegroup acquired during a first time period, the second feature may beacquired based on a second image frame group acquired during a secondtime period, the third feature may be acquired based on a third imageframe group acquired during a third time period, and the first imageframe group, the second image frame group, and the third image framegroup may include a plurality of image frames in common to acquire theoxygen saturation level, the heart rate, and the blood pressure inassociation with each other.

Here, the first color channel value may be a green channel value, thesecond color channel value may be a red channel value, and the thirdcolor channel value may be a blue channel value.

Here, the first feature may be acquired based on the second colorchannel value for a second color channel in which the absorbance ofoxyhemoglobin is lower than the absorbance of hemoglobin and the thirdcolor channel value for a third color channel in which the absorbance ofoxyhemoglobin is higher than the absorbance of hemoglobin.

Here, the second feature may be acquired based on the first differenceand a third difference, which is a difference between the first colorchannel value and the second color channel value for the first region.

Here, in order to reduce noise caused by external light, the first colorchannel value may be a green channel value, the second color channelvalue may be a red channel value, and the third color channel value maybe a blue channel value.

Here, the second feature may include a frequency component value oftime-series data acquired based on the first difference.

Here, the third feature may include a pulse transit time (PTT) acquiredbased on the first difference and the second difference.

Here, the fourth feature may be acquired based on the first difference,the second difference, the third difference, and a fourth difference,which is a difference between the first color channel value and thethird color channel value for the second region.

Here, the first region and the second region may include a face regionof the subject, and the center of the second region may have a differentvertical position from the center of the first region.

According to another embodiment, there may be provided a method ofmeasuring various physiological parameters including a heart rate, anoxygen saturation level, and a blood pressure at the same time, themethod including acquiring a plurality of image frames for a subject,setting a first region, a second region, and a third region for at leastone image frame included in the plurality of image frames, determiningan oxygen saturation level of the subject on the basis of a firstfeature acquired based on at least two of a first color channel value, asecond color channel value, and a third color channel value for thefirst region, determining a heart rate of the subject on the basis of asecond feature acquired based on a first difference, which is adifference between the first color channel value and the second colorchannel value for the first region, determining a blood pressure of thesubject on the basis of a third feature acquired based on a seconddifference, which is a difference between a first color channel valueand a second color channel value for the second region, and a thirddifference, which is a difference between a first color channel valueand a second color channel value for the third region, and outputtingthe oxygen saturation level, the heart rate, and the blood pressure,wherein the first feature may be acquired based on a first image framegroup acquired during a first time period, the second feature may beacquired based on a second image frame group during a second timeperiod, the third feature may be acquired based on a third image framegroup during a third time period, and the first image frame group, thesecond image frame group, and the third image frame group may include aplurality of image frames in common to acquire the oxygen saturationlevel, the heart rate, and the blood pressure in association with eachother.

Here, the first feature may be acquired based on the second colorchannel value for a second color channel in which the absorbance ofoxyhemoglobin is lower than the absorbance of hemoglobin and the thirdcolor channel value for a third color channel in which the absorbance ofoxyhemoglobin is higher than the absorbance of hemoglobin.

Here, the second feature may be acquired based on the first differenceand a third difference, which is a difference between the first colorchannel value and the second color channel value for the first region.

Here, the third feature may include a pulse transit time (PTT) acquiredbased on the first difference and the second difference.

According to still another embodiment, there may be a method ofmeasuring various physiological parameters including a heart rate, anoxygen saturation level, and a blood pressure at the same time, themethod including acquiring a plurality of image frames for a subject,acquiring a first color channel value, a second color channel value, anda third color channel value for at least one image frame included in theplurality of image frames, determining an oxygen saturation level of thesubject on the basis of a first feature acquired based on at least twoof the first color channel value, the second color channel value, andthe third color channel value, determining a heart rate of the subjecton the basis of a second feature acquired based on a first difference,which is a difference between the first color channel value and thesecond color channel value, and a second difference, which is adifference between the first color channel value and the third colorchannel value, determining a blood pressure of the subject on the basisof a third feature acquired based on the first difference and the seconddifference, and outputting the oxygen saturation level, the heart rate,and the blood pressure, wherein the first feature may be acquired basedon a first image frame group acquired during a first time period, thesecond feature may be acquired based on a second image frame groupacquired during a second time period, the third feature may be acquiredbased on a third image frame group acquired during a third time period,and the first image frame group, the second image frame group, and thethird image frame group may include a plurality of image frames incommon to acquire the oxygen saturation level, the heart rate, and theblood pressure in association with each other.

Here, the first feature may be acquired based on the second colorchannel value for a second color channel in which the absorbance ofoxyhemoglobin is lower than the absorbance of hemoglobin and the thirdcolor channel value for a third color channel in which the absorbance ofoxyhemoglobin is higher than the absorbance of hemoglobin.

Here, the second feature may include a frequency component value oftime-series data acquired based on the first difference and the seconddifference.

Here, the third feature may include at least one of a gradient componentvalue, a maximum value, a minimum value, a local maximum value, a localminimum value, the mean of local maximum values, and the mean of localminimum values, a difference between the mean of local maximum valuesand the mean of local minimum values, and an average value oftime-series data acquired based on the first difference and the seconddifference.

According to still another embodiment, there may be provided a method ofmeasuring various physiological parameters including a heart rate, anoxygen saturation level, a blood pressure, and a core temperate at thesame time, the method including acquiring a plurality of image framesfor a subject, setting at least two regions for at least one image frameincluded in the plurality of image frames, acquiring a first colorchannel value, a second color channel value, a third color channelvalue, a fourth color channel value, and a fifth color channel value forat least one image frame included in the plurality of image frames,determining an oxygen saturation level of the subject on the basis of afirst feature acquired based on at least two of the first color channelvalue, the second color channel value, and the third color channelvalue, determining a heart rate of the subject on the basis of a secondfeature acquired based on at least two of the first color channel value,the second color channel value, and the third color channel value,determining a blood pressure of the subject on the basis of a thirdfeature acquired based on at least two of the first color channel value,the second color channel value, and the third color channel value,determining a core temperature of the subject on the basis of a fourthfeature acquired based on the fourth color channel value, and outputtingthe oxygen saturation level, the heart rate, the core temperature, andthe blood pressure, wherein the first color channel value may be a greenchannel value, the second channel value may be a red channel value, thethird color channel value may be a blue channel value, the fourth colorchannel value may be a saturation channel value, and the fifth colorchannel value may be a hue channel value.

Here, the first feature may be acquired based on the second colorchannel value for a second color channel in which the absorbance ofoxyhemoglobin is lower than the absorbance of hemoglobin and the thirdcolor channel value for a third color channel in which the absorbance ofoxyhemoglobin is higher than the absorbance of hemoglobin.

Here, the second feature may be acquired based on a first difference,which is a difference between the first color channel value and thesecond color channel value, and a second difference, which is adifference between the first color channel value and the third colorchannel value.

Here, the second feature may include a frequency component value oftime-series data acquired based on the first difference and the seconddifference.

Here, the third feature may be acquired based on a first difference,which is a difference between the first color channel value and thesecond color channel value, and a second difference, which is adifference between the first color channel value and the third colorchannel value.

Here, the third feature may include at least one of a gradient componentvalue, a maximum value, a minimum value, a local maximum value, a localminimum value, the mean of local maximum values, and the mean of localminimum values, a difference between the mean of local maximum valuesand the mean of local minimum values, and an average value oftime-series data acquired based on the first difference and the seconddifference.

Here, the fourth feature may be acquired based on the fourth colorchannel value and the fifth color channel value.

Here, the fourth feature may include a skin temperature of the subject.

According to still another embodiment, there may be provided a method ofmeasuring various physiological parameters including a heart rate, anoxygen saturation level, and a blood pressure at the same time, themethod including acquiring a plurality of image frames for a subject,acquiring N preliminary heart rates on the basis of at least one imageframe included in the plurality of image frames, acquiring M preliminaryoxygen saturation levels on the basis of at least one image frameincluded in the plurality of image frames, acquiring K preliminary bloodpressures on the basis of at least one image frame included in theplurality of image frames, acquiring a heart rate on the basis of the Npreliminary heart rates, acquiring an oxygen saturation level on thebasis of the M preliminary oxygen saturation levels, acquiring a bloodpressure on the basis of the K preliminary blood pressures, andoutputting the heart rate, the oxygen saturation level, and the bloodpressure, wherein when an image frame acquired when the subject is in afirst state is a first image frame, the first image frame may beincluded in common in the image frames used to obtain the N preliminaryheart rates, the M preliminary oxygen saturation levels, and the K bloodpressures.

According to still another embodiment, there may be provided a method ofacquiring physiological information using various physiologicalparameters, the method including acquiring a plurality of image framesfor a subject, acquiring a first physiological parameter on the basis ofa first image frame group including at least one image frame included inthe plurality of image frames, acquiring a second physiologicalparameter on the basis of a second image frame group including at leastone image frame included in the plurality of image frames, acquiringphysiological information on the basis of at least one of the firstphysiological parameter and the second physiological parameter, andoutputting the first physiological parameter, the second physiologicalparameter, and the physiological information, wherein the first imageframe group and the second image frame group may at least partiallyoverlap each other in order to acquire physiological information inresponse to a specific state of the subject.

Here, the method may further include acquiring personal statistical datafor the subject, and the physiological information may be acquired basedon the first physiological parameter, the second physiologicalparameter, and the acquired personal statistical data.

Here, the first physiological parameter and the second physiologicalparameter may include at least one of a heart rate, an oxygen saturationlevel, a blood pressure, and a core temperature, and the physiologicalinformation may include at least one of drowsiness information, stressinformation, excitement information, and emotion information.

According to an embodiment, there may be provided a method of detectingdrowsiness on the basis of a heart rate, the method being performed byat least one processor and including acquiring a heart rate of asubject, acquiring a comparison result obtained by comparing the heartrate to a reference heart rate, acquiring a duration for which the heartrate is less than or equal to the reference heart rate on the basis ofthe comparison result, and a drowsiness detection operation fordetermining a drowsiness state of the subject on the basis of whetherthe duration reaches a reference duration, wherein the referenceduration may include a first reference duration, a second referenceduration, and a third reference duration, the second reference durationmay be longer than the first reference duration, and the third referenceduration may be longer than the second reference duration, and thedrowsiness state of the subject includes a normal state, a firstdrowsiness state, a second drowsiness state, and a third drowsinessstate, and the first drowsiness state may represent a state in which thesubject is more likely to enter a sleep state than the normal state, thesecond drowsiness state may represent a state in which the subject ismore likely to enter a sleep state than the first drowsiness state, andthe third drowsiness state may represent a state in which the subject ismore likely to enter a sleep state than the second drowsiness state.

Here, the drowsiness detection operation may include determining thatthe drowsiness state of the subject is the first drowsiness state whenthe duration is longer than the first reference duration and shorterthan the second reference duration, determining that the drowsinessstate of the subject is the second drowsiness state when the duration islonger than the second reference duration and shorter than the thirdreference duration, and determining that the drowsiness state is thesecond drowsiness state when the duration is longer than the thirdreference duration.

Here, the drowsiness detection operation may further include determiningthat the drowsiness state of the subject is the normal state when theduration for which the heart rate of the subject is less than or equalto the reference heart rate is shorter than the first referenceduration.

Here, the first drowsiness state may represent a state in which thesubject is not aware of drowsiness but physically likely to enter asleep state.

Here, the heart rate of the subject may represent the mean of aplurality of heart rates during a predetermined time period including atime point at which the heart rate is acquired.

Here, the method may further include acquiring a recovery duration forwhich the heart rate is greater than or equal to the reference heartrate when the drowsiness state of the subject is one of the first tothird drowsiness states and a recovery detection operation ofdetermining the drowsiness state of the subject on the basis of whetherthe recovery duration reaches a reference recovery duration.

Here, the reference recovery duration may include a first referencerecovery duration, a second reference recovery duration, and a thirdreference recovery duration. When it is assumed that the drowsinessstate of the subject is the third drowsiness state, the recoverydetection operation may include determining that the drowsiness state ofthe subject is the second drowsiness state when the recovery duration islonger than the first reference recovery duration and shorter than thesecond reference recovery duration, determining that the drowsinessstate of the subject is the first drowsiness state when the recoveryduration is longer than the second reference recovery duration andshorter than the third reference recovery duration, and determining thatthe drowsiness state of the object is the normal state when the recoveryduration is longer than the third reference recovery duration. When itis assumed that the drowsiness state of the subject is the seconddrowsiness state, the recovery detection operation may includedetermining that the drowsiness state of the subject is the firstdrowsiness state when the recovery duration is longer than the firstreference recovery duration and shorter than the second referencerecovery duration, determining that the drowsiness state of the subjectis the normal state when the recovery duration is longer than the secondreference recovery duration and shorter than the third referencerecovery duration. When it assumed that the drowsiness state of thesubject is the first drowsiness state, the recovery detection operationmay include determining that the drowsiness state of the object is thenormal state when the recovery duration is longer than the firstreference recovery duration and shorter than the second referencerecovery duration.

Here, one of the first reference duration, the second referenceduration, and the third reference duration may be the same as at leastone of the first reference recovery duration, the second referencerecovery duration, and the third reference recovery duration.

Here, the first reference duration, the second reference duration, andthe third reference duration may be different from the first referencerecovery duration, the second reference recovery duration, and the thirdreference recovery duration.

According to an embodiment, there may be provided a method of detectingdrowsiness based on a heart rate and a low frequency (LF)/high frequency(HF) ratio, the method being performed by at least one processor andincluding acquiring a heart rate of a subject, acquiring a comparisonresult obtained by comparing the heart rate to a reference heart rate,acquiring a duration for which the heart rate is changed on the basis ofthe comparison result, acquiring a first drowsiness parameter (adrowsiness level determined based on a heart rate) on the basis of theduration, acquiring an LF/HF ratio of the subject representing a ratioof a sympathetic nerve activity and a parasympathetic nerve activity ofthe subject, acquiring a second drowsiness parameter on the basis of theLF/HF ratio of the subject, and a drowsiness detection operation fordetermining the drowsiness state of the subject using at least one ofthe first drowsiness parameter and the second drowsiness parameter.

Here, the acquiring of the duration may include acquiring the durationon the basis of the length of the duration for which the heart rate isless than or equal to the reference heart rate on the basis of thecomparison result.

Here, the drowsiness state may include a first drowsiness state and asecond drowsiness state. The second drowsiness state is a state in whichthe subject is more likely to enter a sleep state than the firstdrowsiness state. The drowsiness state may be determined according to afirst drowsiness parameter acquired based on a result of comparing theduration to a first reference duration and a second reference durationand a second drowsiness parameter acquired based on a result ofcomparing the LF/HF of the subject to a first reference value and asecond reference value. The drowsiness state may be determined as thefirst drowsiness state when the duration is longer than the firstreference duration and the LF/HF of the subject is smaller than thefirst reference value and may be determined as the second drowsinessstate when the duration is longer than the second reference duration orwhen the LF/HF of the subject is smaller than the second referencevalue.

Here, the drowsiness state may further include a third drowsiness state.The first drowsiness state may be a state in which the subject is notaware of drowsiness but physically has a possibility of entering thesleep state, and the second drowsiness state and the third drowsinessstate may be states in which the subject is aware of drowsiness andphysically has a possibility of entering the sleep state. The thirddrowsiness state may be a state in which the subject is more likely toenter the sleep state than the second drowsiness state.

Here, the drowsiness state may include a normal state, a first-leveldrowsiness state, a second-level drowsiness state, and a third-leveldrowsiness state. The first-level drowsiness state may represent a statein which the subject is more likely to enter the sleep state than thenormal. state, the second-level drowsiness state may represent a statein which the subject is more likely to enter the sleep state than thefirst-level drowsiness state, and the third-level drowsiness state mayrepresent a state in which the subject is more likely to enter the sleepstate than the second drowsiness state. As the first drowsinessparameter and the second drowsiness parameter increase, the possibilityof the subject entering the sleep state may increase. When the firstdrowsiness parameter and the second drowsiness parameter have the samevalue, the value may be acquired, and the drowsiness state may bedetermined as one of the first-level drowsiness state, the second-leveldrowsiness state, and the third-level drowsiness state on the basis ofthe value.

Here, the drowsiness state may include a normal state, a first-leveldrowsiness state, a second-level drowsiness state, and a third-leveldrowsiness state. The first-level drowsiness state may represent a statein which the subject is more likely to enter the sleep state than thenormal state, the second-level drowsiness state may represent a state inwhich the subject is more likely to enter the sleep state than thefirst-level drowsiness state, and the third-level drowsiness state mayrepresent a state in which the subject is more likely to enter the sleepstate than the second drowsiness state. As the first drowsinessparameter and the second drowsiness parameter increase, the possibilityof the subject entering the sleep state may increase. When the firstdrowsiness parameter and the second drowsiness parameter have differentvalues, the larger value between the first drowsiness parameter and thesecond drowsiness parameter may be acquired, and the drowsiness statemay be determined as one of the normal state, the first-level drowsinessstate, the second-level drowsiness state, and the third-level drowsinessstate on the basis of the larger value.

Here, the drowsiness state may include a normal state, a first-leveldrowsiness state, a second-level drowsiness state, and a third-leveldrowsiness state. The first-level drowsiness state may represent a statein which the subject is more likely to enter the sleep state than thenormal state, the second-level drowsiness state may represent a state inwhich the subject is more likely to enter the sleep state than thefirst-level drowsiness state, and the third-level drowsiness state mayrepresent a state in which the subject is more likely to enter the sleepstate than the second drowsiness state. As the first drowsinessparameter and the second drowsiness parameter increase, the possibilityof the subject entering the sleep state may increase. When the firstdrowsiness parameter and the second drowsiness parameter have differentvalues, the smaller value between the first drowsiness parameter and thesecond drowsiness parameter may be acquired, and the drowsiness statemay be determined as one of the normal state, the first-level drowsinessstate, the second-level drowsiness state, and the third-level drowsinessstate on the basis of the smaller value.

Here, the drowsiness state may include a normal state, a first-leveldrowsiness state, a second-level drowsiness state, and a third-leveldrowsiness state. The first-level drowsiness state may represent a statein which the subject is more likely to enter the sleep state than thenormal state, the second-level drowsiness state may represent a state inwhich the subject is more likely to enter the sleep state than thefirst-level drowsiness state, and the third-level drowsiness state mayrepresent a state in which the subject is more likely to enter the sleepstate than the second drowsiness state. As the first drowsinessparameter and the second drowsiness parameter increase, the possibilityof the subject entering the sleep state may increase. When the firstdrowsiness parameter and the second drowsiness parameter have differentvalues, an average value of the first drowsiness parameter and thesecond drowsiness parameter may be acquired, and the drowsiness statemay be determined as one of the normal state, the first-level drowsinessstate, the second-level drowsiness state, and the third-level drowsinessstate on the basis of the average value.

Here, the method may be performed through a recording medium on which aprogram for performing the method is recorded.

According to still another embodiment, there may be provided a smartmirror device configured to display at least two physiologicalparameters among a heart rate, an oxygen saturation level, a bloodpressure, and a core temperature at the same time, the smart mirrordevice including a reflective mirror surface, an image acquisition unitfor acquiring a plurality of image frames for a subject, a display unitplaced behind the reflective mirror surface and configured to displayvisual information through the reflective mirror surface, and a controlunit configured to control the operation of the image acquisition unitand the display unit and acquire a physiological parameter in acontactless manner, wherein the control unit may control the displayunit so that a first physiological parameter acquired based on a firstimage frame group included in the plurality of image frames at a firsttime point is displayed and control the display unit so that a secondphysiological parameter acquired based on a second image frame groupincluded in the plurality of image frames at a second time point isdisplayed, the first image frame group and the second image frame groupmay include at least one image frame in common to associate the firstphysiological parameter and the second physiological parameter with eachother, and the at least one image frame included in the first imageframe group and the second image frame group in common may include animage frame acquired in a first state of the subject observed by thesubject through the reflective mirror surface before the first timepoint and the second time point.

Here, the first physiological parameter and the second physiologicalparameter may include at least one physiological parameter among a heartrate, an oxygen saturation level, a blood pressure, and a coretemperature.

Here, the first image frame group may be identical to the second imageframe group.

Here, the first image frame group may be different from the second imageframe group.

Here, the first time point and the second time point may be the sametime point.

Here, the first time point may be earlier than the second time point,and the at least one image frame included in the first image frame groupand the second image frame group in common may include an image frameacquired in a second state of the subject which is observed by thesubject through the reflective mirror surface before the first timepoint.

Here, the control unit may control the display unit so that a thirdphysiological parameter acquired based on a third image frame groupincluded in the plurality of image frames at a third time point isdisplayed, the first image frame group, the second image frame group,and the third image frame group may include at least one image frame incommon to associate the first physiological parameter, the secondphysiological parameter, and the third physiological parameter with eachother, and at least one image frame included in the first image framegroup, the second image frame group, and the third image frame group incommon may include an image frame acquired in a first state of thesubject which is observed by the subject through the reflective mirrorsurface before; the first time point, the second time point, and thethird time point.

Here, the control unit may acquire the first physiological parameter andthe second physiological parameter on the basis of at least some of theplurality of image frames during a first time period and recognize thesubject on the basis of at least one of the plurality of image framesduring a second time period. The second time period may be shorter thanthe first time period and may be included in the first time period. Thecontrol unit may display information on the recognized subject beforethe first physiological parameter and the second physiological parameterare displayed.

Here, the information on the subject may be the first physiologicalparameter and the second physiological parameter which are previouslymeasured and stored.

Here, the control unit may control the display unit so that firstinformation is displayed at a fourth time point and control the displayunit so that second information is displayed at a fifth time point. Thefirst information may include weather information and time information,the second information may include the information on the subject, thefourth time point may be earlier than the fifth time point, and thefifth time point may be earlier than the first time point and the secondtime point.

Here, the second information may include at least one of scheduleinformation, medication information, recognition information, messengerinformation, and information of interest of the subject.

Here, the control unit may control the display unit so that the firstinformation is displayed at the fourth time point, control the displayunit so that the first information and the second information aredisplayed at the fifth time point, and control the display unit so thatthe first information, the second information, the first physiologicalparameter, and the second physiological parameter are displayed at thesecond time point.

Here, the control unit may operate to acquire the first physiologicalparameter and the second physiological parameter on the basis of atleast three color channel values for the plurality of image frames andmay operate so that the first physiological parameter and the secondphysiological parameter are displayed within 10 seconds after a firstimage frame included in the first image frame group is acquired.

Here, when an image of a plurality of people is included in an imageframe acquired through the image acquisition unit, the control unit mayacquire the first physiological parameter and the second physiologicalparameter on the basis of an image of one person selected from among theplurality of people according to priority and may control the displayunit so that the acquired first and second physiological parameters aredisplayed and so that information on the person selected from among theplurality of people is displayed.

Here, when an image frame including a first subject and a second subjectis acquired after an image frame including the first subject isacquired, the control unit may acquire the first physiological parameterand the second physiological parameter on the basis of an image of thefirst subject and may control the display unit so that information onthe first subject is displayed.

Here, the control unit may prioritize the plurality of people on thebasis of the acquired image frame.

Here, when an image of the first subject and the second subject isincluded in an image frame acquired through the image acquisition unit,the control unit may operate to acquire a first physiological parameterof the first subject on the basis of a fourth image frame group includedin the plurality of image frames, acquire a second physiologicalparameter of the first subject on the basis of a fifth image frame groupincluded in the plurality of image frames, acquire a first physiologicalparameter of the second subject on the basis of a sixth image framegroup included in the plurality of image frames, and acquire a secondphysiological parameter of the second subject on the basis of a seventhimage frame group included in the plurality of image frames. The fourthimage frame group and the fifth image frame group may include at leastone image frame in common, and the sixth image frame group and theseventh image frame group may include at least one image frame incommon.

Here, the control unit may acquire physiological information on thebasis of the first physiological parameter and control the display unitso that the physiological information is displayed at a third timepoint. The physiological information displayed at the third time pointmay be acquired based on the first physiological parameter displayed atthe first time point in order to display the first physiologicalparameter and the physiological information in real time, and the firsttime point may be earlier than the third time point.

Here, the control unit may acquire the physiological information on thebasis of the first physiological parameter and the second physiologicalparameter. The physiological information displayed at the third timepoint may be acquired based on the first physiological parameterdisplayed at the first time point and the second physiological parameterdisplayed at the second time point in order to increase the accuracy ofthe physiological information. The third time point may be later thanthe first time point and the second time point.

Here, the physiological information may include at least one ofcondition information, concentration information, drowsinessinformation, and emotion information.

Here, the control unit may acquire physiological information on thebasis of the first physiological parameter and control the display unitso that the physiological information is displayed at a third timepoint. The physiological information displayed at the third time pointmay be acquired based on first physiological parameters displayed up tothe third time point in order to accurately display the physiologicalinformation, and the third time point may be later than the first timepoint.

Here, the physiological information displayed at the third time pointmay be acquired on the basis of an average value of the firstphysiological parameters displayed up to the third time point.

Here, the control unit may acquire the physiological information on thebasis of the first physiological parameter and the second physiologicalparameter, and the physiological information displayed at the third timepoint may be acquired based on first physiological parameters displayedup to the third time point and second physiological parameters displayedup to the third time point. The third time point may be later than thefirst time point and the second time point.

Here, a cycle in which the first physiological parameter is updated maybe different from a cycle in which the second physiological parameter isupdated.

Here, the control unit may determine an information provision situation,control the display unit so that first information is displayed in thecase of a first situation, and control the display unit so that secondinformation is displayed in the case of a second situation. The firstinformation may include the first physiological parameter and the secondphysiological parameter. The second information may not include thefirst physiological parameter and the second physiological parameter.

Here, the control unit may determine the information provision situationon the basis of a movement direction of the subject.

Here, the smart mirror device may further include an operation detectionsensor for detecting the direction of the movement of the subject.

Here, the control unit may determine the information provision situationon the basis of the plurality of acquired image frames.

Here, the first information may include at least one of external weatherinformation, schedule information of the subject, and time information,and the second information may include at least one of internaltemperature information, internal humidity information, internal airinformation, security information, and activity time information.

Here, the first information may include at least two physiologicalparameters, and the second information may include at least one ofweather information, time information, news information, scheduleinformation, and medication information.

Here, the control unit may acquire at least three color channel valuesfor at least one image frame included in the plurality of image frames,acquire a first difference and a second difference on the basis of theat least three color channel values, and acquire the first physiologicalparameter and the second physiological parameter on the basis of thefirst difference and the second difference.

According to still another embodiment, there may be provided a method ofoperating a smart mirror device configured to display at least twophysiological parameters among a heart rate, an oxygen saturation level,a blood pressure, and a core temperature at the same time, the methodincluding acquiring an on-trigger, acquiring a plurality of image framesfor a subject, acquiring a first physiological parameter on the basis ofa first image frame group included in the plurality of image frames,acquiring a second physiological parameter on the basis of a secondimage frame group included in the plurality of image frames, displayingthe first physiological parameter and the second physiologicalparameter, acquiring an off-trigger, and stopping the acquisition of theplurality of image frames for the subject, wherein the first image framegroup and the second image frame group may include at least one imageframe in common to associate the first physiological parameter and thesecond physiological parameter with each other, the on-trigger may beacquired from at least one sensor, and the off-trigger may be acquiredfrom an image sensor for acquiring the plurality of image frames.

Here, the on-trigger may be acquired from at least one of an operationdetection sensor, a touch sensor, a mouse, and a keyboard.

According to still another embodiment, there may be provided a smartmirror device configured to display at least one physiological parameteramong a heart rate, an oxygen saturation level, a blood pressure, and acore temperature, the smart mirror device including a reflective mirrorsurface, an image acquisition unit for acquiring a plurality of imageframes, a display unit placed behind the reflective mirror surface andconfigured to display visual information through the reflective mirrorsurface, a switch unit placed in front of the image acquisition unit andconfigured to switch a field of view of the image acquisition unit, anda control unit configured to control operations of the image acquisitionunit and the display unit and acquire a physiological parameter, whereinthe switch unit may have a surface formed as a reflective mirror, andwhen the switch unit is open, the control unit may control the displayunit so that the physiological parameter and at least one piece ofvisual information are displayed through the display unit, and when theswitch unit is closed, the control unit may control the display unit sothat at least one piece of visual information is displayed through thedisplay unit.

Here, when the switch unit is changed from the closed state to the openstate, the control unit may control the image acquisition unit so thatan image frame is acquired from the image acquisition unit, and when theswitch unit is changed from the open state to the closed state, thecontrol unit may control the image acquisition unit so that the imageacquisition unit cannot acquire the image frame.

0. Definition of Terms

The term “measurement” used herein can be understood as a conceptincluding direct measurement, speculative measurement, and measurementof amounts relative to a certain amount.

The term “heart rate” used herein can be understood as a heart rate,which can be understood as a concept including both of a heart rate,which may refer to the number of beats measured near a heart due toheartbeats, and a pulse rate, which may refer to vibration generated dueto heartbeats and propagated to peripheral blood vessels.

The term “blood pressure” used herein can be understood as a pressuregenerated in blood vessels when blood is pushed from a heart and can beunderstood as a value that can be estimated as a value that can beunderstood as typical blood pressure regardless of the measurement site(e.g., a value measured in the artery in an upper arm).

The terms “oxygen saturation level” used herein can be understood as thedegree of saturation of oxygen in blood, and more specifically, it canbe understood as the fraction of oxyhemoglobin to total hemoglobin inblood.

The term “core temperature” used herein can be understood as the bodytemperature of people or animals and may be different from skintemperature which may be measured on skin.

The term “skin temperature” used herein can be understood as the surfacetemperature of skin being measured.

The term “image” used herein can be understood as a concept includingone image or a plurality of images included in a video.

The term “color channel” used herein can be understood as each axisconstituting color space. For example, a red channel, a green channel,and a blue channel may refer to a red axis, a green axis, and a blueaxis constituting RGB color space. Such a color channel may be formed intwo dimensions, three dimensions, or four dimensions.

The term “image of a subject” used herein can be understood as an imageincluding a measurement position of a subject. For example, when themeasurement position is a subject's face, it can be understood as animage including a face region of the subject.

The term “personal statistical data” used herein may refer tocollectible personal statistical data of a subject such as age, gender,height, and weight, observable personal statistical data such as facialexpressions, wrinkles, and face color of a subject, or quantifiablepersonal statistical data such as statistical data the average bloodpressure of people in their 20s, the average skin color of Asian people,the average height of men in their 30s, the average weight of Koreanmen, etc.) calculated for a group including or relating to a subject.

The term “time series data” used herein may refer to data listed along atime axis, but the present invention is not limited thereto. This termmay refer to data listed along an image frame axis that may correspondto time, may refer to data that can be sorted along a time axis or animage frame axis, and can be understood as typical time series data.

1. Physiological-Parameter and Physiological-Information ManagementSystem

The term “physiological parameter” may refer to a result of a measurableor estimable human physiological activity, for example, heart rate,oxygen saturation level, blood pressure, core temperature, blood flow,etc. However, the present invention is not limited thereto, and thisterm may refer to other results of measurable or estimable humanphysiological activities.

The term “physiological information” may refer to information that maybe calculated in consideration of at least some results of humanphysiological activities such as a physiological parameter and personalstatistical data such as facial expressions, postures, and age and mayinclude drowsiness information, stress information, excitementinformation, emotion information, etc., but the present invention is notlimited thereto.

The term “physiological-parameter and physiological-informationmanagement system” may refer to a management system capable of storing,sharing, or analyzing measurable or estimable physiological parametersand calculable physiological information and comprehensively managingindividuals' health or the like by using the physiological parametersand physiological information.

FIGS. 1 and 2 are diagrams showing a physiological-parameter andphysiological-information management system according to an embodiment.

Referring to FIG. 1, a physiological-parameter andphysiological-information management system 100 according to anembodiment may include a physiological parameter acquisition device 10and a server 20.

In this case, the physiological parameter acquisition device 10 maymeasure a physiological parameter of a subject. In detail, thephysiological parameter acquisition device 10 may measure aphysiological parameter of a subject in an invasive manner, in anon-invasive and contact manner, or in a contactless manner.

For example, the physiological parameter acquisition device 10 mayanalyze a video or image of the subject and measure a physiologicalparameter, such as a heart rate, an oxygen saturation level, and a bloodpressure, of the subject, but the present invention is not limitedthereto.

Also, the physiological parameter acquisition device 10 may calculatephysiological information on the basis of the measured physiologicalparameter. In detail, the physiological parameter acquisition device 10may calculate physiological information on the basis of the measuredphysiological parameter and may calculate physiological information inconsideration of the measured physiological parameter and personalstatistical data such as a facial expression, a posture, and an age.

For example, the physiological parameter acquisition device 10 maycalculate physiological information, such as emotion information anddrowsiness information, of the subject on the basis of the measuredphysiological parameter such as a heart rate, an oxygen saturationlevel, and a blood pressure, but the present invention is not limitedthereto.

Also, the physiological parameter acquisition device 10 may store themeasured physiological parameter and the calculated physiologicalinformation, For example, the physiological parameter acquisition device10 may store the measured physiological parameter and the calculatedphysiological information of the subject in an internal memory, but thepresent invention is not limited thereto.

Also, the physiological parameter acquisition device 10 may display themeasured physiological parameter and the calculated physiologicalinformation. For example, the physiological parameter acquisition device10 may further include a display and may display the measuredphysiological parameter and the calculated physiological information ofthe subject using the display, but the present invention is not limitedthereto. The physiological parameter acquisition device 10 may transmitthe corresponding information to an external display so that theinformation can be displayed through the external display.

Also, the physiological parameter may be displayed once through thedisplay, and a physiological parameter changing in real time may becontinuously displayed.

Also, the physiological parameter acquisition device 10 may transmit thephysiological parameter and the physiological information of the subjectto the server 20.

In this case, the physiological parameter and the physiologicalinformation transmitted to the server 20 may be stored in the server 20as personal data. For example, a measured physiological parameter andcalculated physiological information of a subject A may be stored in theserver 20 as data on the subject A, and a measured physiologicalparameter and calculated physiological information of a subject B may bestored in the server 20 as data on the subject B.

Also, when necessary, the server 20 may communicate with an externalterminal to transmit a physiological parameter and physiologicalinformation of a subject. For example, when the subject A whosephysiological parameter and physiological information are stored in theserver 20 visits a hospital to receive treatment, a doctor who willtreat the subject A may need the physiological parameter andphysiological information of the subject A. In this case, when theserver 20 receives a request for transmission of the physiologicalparameter and physiological information of the subject A from anexternal terminal placed in the hospital, the server 20 may communicatewith the external terminal to transmit the physiological parameter andphysiological information of the subject A.

As described above, the physiological-parameter andphysiological-information management system 100 may serve as a basis forproviding a continuous and comprehensive management service for anindividual's health. In addition to the above example, it is obviousthat the physiological-parameter and physiological-informationmanagement system 100 can serve as a basis for providing variouscomprehensive management services using physiological parameters andphysiological information which are continuously measured, stored, andmanaged.

Also, referring to FIG. 2, a physiological-parameter andphysiological-information management system 100 according to anembodiment may include an image acquisition device 30 and a server 20.

In this case, the image acquisition device 30 may acquire a video orimage of a subject.

Also, the server 20 may acquire a video or image of a subject from theimage acquisition device 30.

Also, the server 20 may acquire personal statistical data of the subjectfrom an input device, an external terminal, or the like.

Also, the server 20 may measure a physiological parameter of the subjecton the basis of the acquired video or image. For example, the server 20may analyze the acquired video or image to measure the heart rate,oxygen saturation level, blood pressure, and the like of the subject,but the present invention is not limited thereto.

Also, the server 20 may calculate physiological information on the basisof the measured physiological parameter. In detail, the server 20 maycalculate the physiological information on the basis of the measuredphysiological parameter and may calculate the physiological informationin comprehensive consideration of the measured physiological parameterand personal statistical data such as facial expression, posture, andage.

For example, the server may calculate the physiological information,such as emotion information and drowsiness information, of the subjecton the basis of the measured physiological parameter such as the heartrate, the oxygen saturation level, and the blood pressure, but thepresent invention is not limited thereto.

Also, the server 20 may store the measured physiological parameter andthe calculated physiological information.

Also, since it is apparent that the physiological-parameter andphysiological-information management system 100 including the imageacquisition device 30 and the server 20 can perform the functions of thephysiological-parameter and physiological-information management systemthat has been described with reference to FIG. 1, a redundantdescription thereof will be omitted here.

2. Various Embodiments of Physiological Parameter Acquisition Device

FIG. 3 is a diagram illustrating a physiological parameter acquisitiondevice according to an embodiment.

Referring to FIG. 3, a physiological parameter acquisition device 1000according to an embodiment may include an image acquisition unit 1010, acontrol unit 1020, a storage unit 1030, and a communication unit 1040.However, the physiological parameter acquisition device 1000 may includeat least some of the image acquisition unit 1010, the control unit 1020,the storage unit 1030, and the communication unit 1040. For example, thephysiological parameter acquisition device 1000 may include only theimage acquisition unit 1010 and the control unit 1020. However, thepresent invention is not limited thereto and may be implemented invarious ways.

Also, the image acquisition unit 1010 may acquire a video or image of asubject. In detail, the image acquisition unit 1010 may include aphotographing device and may acquire a video or image of the subjectusing the photographing device or a photographing device placed outsidethe physiological parameter acquisition device 1000, but the presentinvention is not limited thereto.

Also, when the image acquisition unit 1010 acquires a video or image ofa subject from the photographing device, the photographing device may beprovided as a visible camera for acquiring a visible light image, aninfrared (IR) camera for acquiring an infrared image, and the like.However, the present invention is not limited thereto, and a hybrid-typecamera for acquiring a visible light image and an infrared image may beprovided.

Also, when the photographing device acquires a visible light image, theacquired visible light image may be acquired as at least one colorchannel value. For example, the acquired visible light image may beacquired as a color channel value of an RGB color space, which isrepresented using red, green, and blue or a color channel value of anHSV color space, which is represented using hue, saturation, andbrightness (value). However, the present invention is not limitedthereto, and the acquired visible light image may be acquired as colorchannel values of various color spaces such as YCrCb and YiQ.

Also, when the photographing device acquires an infrared image, thephotographing device may acquire an infrared image through an infraredlight source placed inside or outside the photographing device. In thiscase, the infrared light source may emit near-infrared light in awavelength range of 750 nm to 3000 nm. However, the present invention isnot limited thereto, and the infrared light source may emitmiddle-infrared light, far-infrared light, and extreme-infrared light.

Also, the control unit 1020 may acquire a physiological parameter usingan image of a subject acquired from the image acquisition unit 1010.

For example, the control unit 1020 may analyze the image of the subjectacquired from the image acquisition unit 1010 to acquire a physiologicalparameter, such as a heart rate, an oxygen saturation level, a bloodpressure, and a core temperature, of the subject. However, the presentinvention is not limited thereto, and the control unit 1020 may acquirevarious physiological parameters.

Also, the control unit 1020 may calculate physiological information onthe basis of the acquired physiological parameter.

For example, the control unit 1020 may calculate physiologicalinformation, such as emotion information and drowsiness information, ofthe subject on the basis of the acquired physiological parameter such asthe heart rate, the oxygen saturation level, the blood pressure, and thecore temperature. However, the present invention is not limited thereto,and the control unit 1020 may calculate various pieces of physiologicalinformation.

Also, the control unit 1020 may control the operation of at least someof the image acquisition unit 1010, the storage unit 1030, and thecommunication unit 1040.

Also, the storage unit 1030 may store the physiological parameter andphysiological information acquired by the control unit 1020. In detail,the storage unit 1030 may store a physiological parameter andphysiological information of one subject and also may store aphysiological parameter and physiological information of each of severalsubjects.

Also, the communication unit 1040 may transmit the physiologicalparameter and physiological information acquired by the control unit1020. In detail, the communication unit 1040 may transmit thephysiological parameter and physiological information acquired by thecontrol unit 1020 to a management server or to a user terminal.

3. Various Embodiments of Method of Acquiring Physiological Parameterand Physiological Information 3.1 Physiological Parameter AcquisitionMethod

FIG. 4 is a flowchart showing a physiological parameter acquisitionmethod according to an embodiment.

Referring to FIG. 4, a physiological parameter acquisition method 1100according to an embodiment may include acquiring an image of a subject(S1110).

In this case, as described above, the image of the subject may beacquired using various cameras such as a visible light camera and aninfrared light camera or using various other cameras, and thus adetailed description thereof will be omitted.

Also, the physiological parameter acquisition method 1100 according toan embodiment may include detecting a skin region (S1120).

In this case, the skin region may refer to a region that can beestimated as a skin region of the subject in the image of the subject.

Also, the skin region may well reflect changes in blood vessels due toheartbeats. The detection of the skin region in this way can increasethe accuracy of the acquisition of the physiological parameter.

Also, according to an embodiment, in order to detect the skin region,non-skin regions such as the eyes and hair of the subject rather thanthe skin region, which can reflect a change in color due to the dilationof blood vessels, may be removed. For example, the hue values of thenon-skin regions such as the eyes and hair of the subject may bereplaced with a meaningless value such as black, but the presentinvention is not limited thereto.

Also, according to an embodiment, a specific color space may be used todetect the skin region. For example, the detecting of the skin region(S1120) may include replacing the acquired image of the subject with avalue in the YCrCb color space and detecting the skin region on thebasis of an image represented in the YCrCb color space, but the presentinvention is not limited thereto.

Also, it is obvious that the skin region can be detected using variouswell-known techniques.

Also, the physiological parameter acquisition method 1110 according toan embodiment may include setting a region of interest (ROI) (S1130).

In this case, the ROI may refer to a region of interest for dataprocessing in the acquired image of the subject and also may refer to aregion to be used for data processing so as to acquire a physiologicalparameter, but the present invention is not limited thereto.

Also, according to an embodiment, in order to set the ROI, a face regionof the subject may be set. For example, in order to set an ROI includedin the subject's face, a face region of the subject may be set.

In detail, a region having a certain proportion from the center of theset face region of the subject may be set as an ROI.

For example, a region corresponding to 80% in length and 60% in width onthe basis of the center of the face region of the subject may becropped, but the present invention is not limited thereto.

Also, according to an embodiment, a feature point may be used to set theROI. For example, a feature point of a nose region of the subject may beextracted from the acquired image of the subject, and an ROI may be setbased on the extracted feature point. However, the present invention isnot limited thereto.

In detail, a region with a certain size from the center of the extractedfeature point may be set as an ROI in the set face region, but thepresent invention is not limited thereto.

Also, according to an embodiment, a plurality of feature points may beused to set the ROI. For example, feature points of an eye region and anoise region of the subject may be extracted from the acquired image ofthe subject, and the ROI may be set based on the extracted featurepoints.

Also, when an ROI is set for each of a plurality of images that areconsecutively acquired, the ROIs may be set for the plurality of imagesindividually or in association with each other.

For example, in order to set the ROIs of the plurality of images inassociation with each other, a face region of the subject may be set foreach of the plurality of images. When the difference between the centerof a face region set for a first image frame and the center of a faceregion set for a second image frame, which is acquired after the firstimage frame, does not exceed a threshold, the face region of the secondimage frame may be set to be the same as the face region set for thefirst image frame, but the present invention is not limited thereto.

Also, when the difference between the center of the face region set inthe first image frame and the center of the face region set in thesecond image frame exceeds the threshold, the face region of the secondimage frame may be set to be different from the face region set in thefirst image frame, but the present invention is not limited thereto.

Also, an ROI according to an embodiment may be set to include a portionof a body part or a portion of a face depending on a physiologicalparameter to be acquired and may include at least one ROI.

For example, the ROI may be set to include at least a portion of a cheekregion in order to acquire a heart rate as the physiological parameter.In detail, the ROI may be set to include a cheek region which can wellreflect the degree of the dilation of blood vessels according to bloodflow and from which a heart rate is easy to acquire, but the presentinvention is limited thereto.

Also, for example, at least two such ROIs may be set to acquire a bloodpressure as the physiological parameter, and specifically, may includean ROI including an upper face region and an ROI including a lower faceregion in order to reflect blood flow.

Also, for example, at least two such ROIs may be set to acquire a bloodpressure as the physiological parameter, and specifically, may be set astwo or more ROIs spaced different distances from a heart. For example,the ROIs may be set to include an ROI including a hand region and an ROIincluding a face region, but the present invention is not limitedthereto.

Also, the physiological parameter acquisition method 1110 according toan embodiment may include processing data for the ROI (S1140).

Also, a color channel value for the ROI may be extracted to process thedata for the ROI. In this case, the color channel value may be the meanof color channel values of pixels included in the ROI and may bereferred to as an average pixel value.

For example, when color channel values corresponding to the RGB colorspace are extracted, a red channel pixel value, a green channel pixelvalue, and a blue channel pixel value of each pixel included in the ROImay be extracted. A red channel value, which is the mean of red channelpixel values included in the ROI, a blue channel value, which is themean of blue channel pixel values included in the ROI, and a greenchannel value, which is the mean of green channel pixel values includedin the ROI, may be extracted. However, the present invention is notlimited thereto, and color channel values corresponding to various colorspaces such as the HSV color space and the YCrCb color space may beextracted.

Also, a color channel value extracted according to a specific colorspace may be converted into another color space. For example, a colorchannel value extracted according to the RGB color space may beconverted into color channel values corresponding to various colorspaces such as the HSV color space and the YCrCb color space.

A color channel value extracted to process the data for the ROI may be acolor channel value obtained by weighting at least some of the colorchannel values extracted according to various color spaces and combiningthe color channel values.

Also, the color channel value may be extracted for each of a pluralityof image frames that are consecutively acquired or may be extracted fromat least some of the image frames.

Also, color channel values extracted from one image frame may beprocessed through a mathematical operation or the like. In detail, aplurality of channel values acquired from one image frame may beprocessed through a mathematical operation such as addition orsubtraction. For example, a green channel value and a red channel valueacquired from one image frame may be processed through a subtractionoperation. However, the present invention is not limited thereto, andvarious channel values may be processed through various mathematicaloperations.

Also, color channel values extracted from each of the plurality of imageframes or processed values of the color channel values may be processedthrough a mathematical operation or the like. In detail, color channelvalues extracted from each of the plurality of image frames or processedvalues of the color channel values may be processed through amathematical operation such as by obtaining an average or a deviation ina certain section. However, the present invention is not limitedthereto, and the color channel values or the processed values may beprocessed by obtaining the difference between the maximum values andminimum values for a certain section and through various mathematicaloperations.

Also, the color channel values extracted from each of the plurality ofimage frames and the processed values of the color channel values may beprocessed to acquire at least one piece of time-series data.

Also, a characteristic value for acquiring a physiological parameter maybe extracted based on at least some of the color channel values, theprocessed values, and the time-series data. For example, a frequencycomponent for acquiring a heart rate may be extracted based on afrequency component of the time-series data, but the present inventionis not limited thereto.

Also, the physiological parameter acquisition method 1110 according toan embodiment may include acquiring a physiological parameter (S1150).

Also, a characteristic value extracted from the ROI may be used toacquire the physiological parameter. For example, a heart rate based onthe frequency component of the time-series data extracted from the ROImay be acquired, but the present invention is not limited thereto.

Also, the processing of the data for the ROI (S1140) and the acquiringof the physiological parameter (S1150) may vary depending on eachphysiological parameter, and a detailed description thereof will bedescribed in detail in corresponding sections.

FIG. 5 is a diagram showing a physiological parameter acquisition methodaccording to an embodiment.

Referring to FIG. 5, an image 1161 of a subject may include a faceregion. A skin region may be detected (1162), and an ROI may be detected(1163). A detailed description thereof is redundant and will be omitted.

Also, referring to FIG. 5, a color channel value for the ROI may beextracted (1164), the extracted color channel value may be processed(1165), and a physiological parameter may be acquired based on theprocessed color channel value (1166).

However, this will be described in detail in the corresponding sectionbelow.

3.2 Physiological Information Acquisition Method

FIG. 6 is a flowchart showing a physiological information acquisitionmethod according to an embodiment.

Referring to FIG. 6, the physiological information acquisition methodaccording to an embodiment may include acquiring a physiologicalparameter (S1210).

In this case, the physiological parameter may be acquired by theabove-described physiological parameter acquisition method. However, thepresent invention is not limited thereto, and the physiologicalparameter may be acquired by an external sensor such as anelectrocardiography (ECG) sensor.

Also, a redundant description of the physiological parameter will beomitted.

Also, the physiological information acquisition method according to anembodiment may include acquiring personal statistical data (S1220).

In this case, the personal statistical data may refer to collectiblepersonal statistical data, such as age and gender, of the subject andalso may refer to observable personal statistical data, such as facialexpression and wrinkles, of the subject. However, the present inventionis not limited thereto, and the personal statistical data may be varioustypes of personal statistical data other than the physiologicalparameter for acquiring the physiological information.

Also, the physiological information acquisition method according to anembodiment may include acquiring physiological information (S1230).

In this case, the physiological information may be drowsinessinformation, emotion information, or the like of the subject, but thepresent invention is not limited thereto.

Also, the physiological parameter may be used to acquire thephysiological information. For example, a heart rate may be used as aphysiological parameter to acquire the drowsiness information. Indetail, when the heart rate of the subject is less than or equal to areference heart rate, the subject may be regarded as being drowsy. Thedrowsiness level of the subject may be acquired according to a timeperiod for which the heart rate is less than or equal to the referenceheart rate.

Also, for example, a heart rate and a blood pressure may be used asphysiological parameters to acquire the emotion information. In detail,when the heart rate and blood pressure of the subject are greater thanor equal to a reference heart rate and a reference blood pressure, thesubject may be regarded as being excited.

Also, the physiological parameter and the personal statistical data maybe used to acquire the physiological information. For example, a heartrate and personal statistical data, such as age and gender, of thesubject may be used to acquire the drowsiness information.

Also, for example, a heart, rate, a blood pressure, and personalstatistical data, such as facial expression, age, and gender, of thesubject may be used to acquire the emotion information.

Also, the physiological parameter and the personal statistical data maybe weighted to acquire the physiological information. For example, inorder to acquire the physiological information, different weights may beassigned to the physiological parameter and the personal statisticaldata, and different weights may be assigned depending on the subject.

3.3 Physiological Parameter Acquisition Method Using PhysiologicalParameter Acquisition Model

FIGS. 7 and 8 are diagrams illustrating a physiological parameteracquisition method using a physiological parameter acquisition model.

FIG. 7A shows a physiological parameter acquisition method 1300 using aphysiological parameter acquisition model 1302 according to anembodiment.

In this case, the physiological parameter acquisition model 1302according to an embodiment may be implemented in a machine learningmethod. For example, the physiological parameter acquisition model 1302may be a model implemented through supervised learning. However, thepresent invention is not limited thereto, and the physiologicalparameter acquisition model 1302 may be a model implemented throughunsupervised learning, semi-supervised learning, reinforcement learning,and the like.

Also, the physiological parameter acquisition model 1302 according to anembodiment may be implemented as an artificial neural network (ANN). Forexample, the physiological parameter acquisition model 1302 may beimplemented as a feedforward neural network, a radial basis functionnetwork, a Kohonen self-organizing network, or the like, but the presentinvention is not limited thereto.

Also, the physiological parameter acquisition model 1302 according to anembodiment may be implemented as a deep neural network (DNN). Forexample, the physiological parameter acquisition model 1302 may beimplemented as a convolutional neural network (CNN), a recurrent neuralnetwork (RNN), a long short-term memory network (LSTM), or gatedrecurrent units (GRUs), but the present invention is not limitedthereto.

Also, an image 1301 input to the physiological parameter acquisitionmodel 1302 may be acquired image data itself.

Also, an image 1301 input to the physiological parameter acquisitionmodel 1302 may be pre-processed image data. For example, the image 1301may be subject to the Eulerian video magnification. However, the presentinvention is not limited thereto, and various pre-processes such as aprocess of obtaining the mean of acquired RGB values may be performed.

Also, the acquired physiological parameter 1303 may include a heartrate, an oxygen saturation level, a blood pressure, a core temperature,or the like.

Also, the acquired physiological parameter 1303 may have onephysiological parameter or have a plurality of physiological parametersacquired at the same time. For example, a heart rate may be acquired asa result of the physiological parameter acquisition model 1302. However,the present invention is not limited thereto, and both of a heart rateand a blood pressure may be acquired as a result of the physiologicalparameter acquisition model 1302.

Also, FIG. 7B shows a physiological parameter acquisition method 1350using a physiological parameter acquisition model 1354 according toanother embodiment.

In this case, the physiological parameter acquisition model 1354 mayacquire personal statistical data 1353 and a feature 1352 extracted froman image 1351 as an input value. For example, time-series data for acolor channel value may be extracted from the image 1351 as a feature,and the physiological parameter acquisition model 1354 may acquire thetime-series data for the color channel value and personal statisticaldata as an input value and may calculate a physiological parameter 1355as a result.

Also, the personal statistical data may refer to collectible personalstatistical data, such as age, gender, height, and weight, of thesubject, observable personal statistical data, such as facialexpression, wrinkles, and face color, of the subject, or quantifiablepersonal statistical data such as an average blood pressure, an averagecolor, an average height, and an average weight.

Also, the description of the physiological parameter acquisition model1302 may be applied to the physiological parameter acquisition model1354, and thus a redundant description thereof will be omitted.

Also, FIG. 8 shows a physiological parameter acquisition method 1400using a physiological parameter acquisition model 1405 according tostill another embodiment.

In this case, the physiological parameter acquisition method 1400 mayinclude a feature extraction model 1402, and the feature extractionmodel 1402 according to an embodiment may be implemented in a machinelearning method. For example, the feature extraction model 1402 may be amodel implemented through supervised learning. However, the presentinvention is not limited thereto, and the feature extraction model 1402may be a model implemented through unsupervised learning,semi-supervised learning, reinforcement learning, and the like.

Also, the feature extraction model 1402 according to an embodiment maybe implemented as an artificial neural network (ANN). For example, thefeature extraction model 1402 may be implemented as a feedforward neuralnetwork, a radial basis function network, a Kohonen self-organizingnetwork, or the like, but the present invention is not limited thereto.

Also, the feature extraction model 1402 according to an embodiment maybe implemented as a deep neural network (DNN). For example, the featureextraction model 1402 may be implemented as a convolutional neuralnetwork (CNN), a recurrent neural network (RNN), a long short-termmemory network (LSTM), or gated recurrent units (GRUs), but the presentinvention is not limited thereto.

Also, the physiological parameter acquisition model 1405 may acquirepersonal statistical data 1404 and a feature 1403 extracted by thefeature extraction model 1402 as an input value and may calculate aphysiological parameter 1406 as a result on the basis of the inputvalue.

Also, the description of the physiological parameter acquisition model1302 may be applied to the physiological parameter acquisition model1405, and thus a redundant description thereof will be omitted.

Machine learning, an artificial neural network, or a deep neural networkmodel may be used to acquire a physiological parameter like an examplethat has been described with reference to FIGS. 7 and 8.

4. Various Embodiments of Physiological Parameter Acquisition 4.1Various Embodiments of Heart Rate Measurement Method

When a heart beats in the body of a living organism, blood can becarried throughout the body by the heartbeats. In this case, the bloodflows through blood vessels. The volume of the blood vessels may changeover time, and the amount of blood contained in the blood vessels maychange.

Accordingly, when the change in the volume of the blood vessels or thechange in the amount of blood is measured, a heart rate may be acquired.For example, when the amount of blood included in the blood vesselschanges, the amounts of hemoglobin and oxyhemoglobin contained in theblood may change, and thus the amount of light reflected by the bloodmay change. Accordingly, when the change in the amount of lightreflected by the blood is measured, a heart rate may be acquired.

Also, it is obvious that, in addition to the above-described exemplaryprinciple, various principles for measuring a heart rate with light areapplicable.

FIG. 9 is a flowchart illustrating a heart rate measurement methodaccording to an embodiment.

Referring to FIG. 9, a heart rate measurement method 1500 according toan embodiment may include at least some of an operation of acquiring animage for at least one of a plurality of acquired image frames (S1510),an operation of detecting a skin region (S1520), an operation ofdetecting an ROI (S1530), and an operation of processing data for theROI (S1540), but the present invention is not limited thereto.

Also, the operation of acquiring the image (S1510), the operation ofdetecting the skin region (S1520), and the operation of detecting theROI (S1530) have been described above, and thus a redundant descriptionthereof will be omitted.

Also, the operation of processing the data for the ROI (S1540) may beperformed on at least one of a plurality of acquired image frames.

Also, a color channel value for the ROI may be extracted for the atleast one of the plurality of acquired image frames in order to processthe data for the ROI. In this case, the color channel value may be themean of color channel values of pixels included in the ROI and may bereferred to as an average pixel value.

Also, the operation of processing the data for the ROI (S1540) has beendescribed in detail above, and thus a redundant description thereof willbe omitted.

Also, the heart rate measurement method 1500 according to an embodimentmay include at least some of an operation of extracting time-series datafor an image frame group including at least some of the plurality ofacquired image frames (S1550) and an operation of acquiring a heart rate(S1560), but the present invention is not limited thereto.

Also, the operation of extracting the time-series data (S1550) may beperformed on the image frame group including at least some of theplurality of acquired image frames.

In this case, the image frame group may be a group of a plurality ofconsecutive or nonconsecutive image frames. For example, the image framegroup may refer to a group of consecutive image frames starting from afirst image frame up to a 180^(th) image frame. However, the presentinvention is not limited thereto, and the image frame group may refer toa group of at least some of the image frames starting from the firstimage frame up to the 180^(th) image frame.

Also, the operation of acquiring the heart rate (S1560) may be performedon the image frame group including at least some of the plurality ofacquired image frames.

In this case, a frequency component of the acquired time-series data maybe extracted to acquire the heart rate. For example, in order to acquirethe heart rate, the time-series data may be transformed by a Fouriertransform (FT) to extract the frequency component. However, the presentinvention is not limited thereto, and the time-series data may betransformed by a fast Fourier transform (FFT), a discrete Fouriertransform (DFT), the short-time Fourier transform (STFT)), or the like.However, the present invention is not limited thereto, and thetime-series data may be subject to various processes for extractingfrequency components.

Also, the heart rate may be acquired based on one heartbeat and may beacquired based on at least two heartbeats. For example, one heartbeatmay be acquired based on one piece of time-series data, and anotherheart heat may be acquired based on another piece of time-series data. Afinal heartbeat may be acquired based on at least two acquired heartrates, but the present invention is not limited thereto.

4.2 Various Embodiments of Oxygen Saturation Level Measurement Method

Oxygen saturation level (SpO2) may refer to the amount of oxygencombined with hemoglobin and may indicate a ratio of oxyhemoglobin to atotal amount of hemoglobin in blood.

Also, hemoglobin and oxyhemoglobin may have the same or differentabsorbances for light with one wavelength. For example, hemoglobin andoxyhemoglobin may have different absorbances for light in a wavelengthrange of 700 nm, may have different absorbances for light in awavelength range of 1000 nm, and may have similar absorbances for lightin a wavelength range of 800 nm.

Also, when the amount of blood included in blood vessels changes, theamounts of hemoglobin and oxyhemoglobin contained in the blood maychange.

Accordingly, an oxygen saturation level may be acquired by usingextinction coefficients of hemoglobin and oxyhemoglobin for light in afirst wavelength range, extinction coefficients of hemoglobin andoxyhemoglobin for light in a second wavelength range, the degree ofchange of light in the first wavelength range due to a change in theamount of blood, and the degree of change of light in the secondwavelength range due to a change in the amount of blood.

For example, when it is assumed that the extinction coefficient ofoxyhemoglobin for light in the first wavelength range is ε₁, that theextinction coefficient of hemoglobin for light in the first wavelengthrange is ε₂, that the extinction coefficient of oxyhemoglobin for lightin the second wavelength range is ε₃, that the extinction coefficient ofhemoglobin for light in the second wavelength range is ε₄, and that theproportion of oxyhemoglobin is S, Equation 1 may be obtained as follows:

$\begin{matrix}{\frac{{S*ɛ_{1}} + {\left( {1 - S} \right)*ɛ_{2}}}{{S*ɛ_{3}} + {\left( {1 - S} \right)*ɛ_{4}}} = {\frac{A\; {C_{\lambda \; 1}/D}\; C_{\lambda \; 1}}{A\; {C_{\lambda \; 2}/D}\; C_{\lambda \; 2}}.}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, the oxygen saturation level may be represented as S*100(%) but thepresent invention is not limited thereto.

Also, it is obvious that, in addition to the above-described exemplaryprinciple, various principles for measuring oxygen saturation levelswith light are applicable.

FIG. 10 is a flowchart illustrating an oxygen saturation levelmeasurement method according to an embodiment.

Referring to FIG. 10, an oxygen saturation level measurement method 1600according to an embodiment may include at least some of an operation ofacquiring an image for at least one of a plurality of acquired imageframes (S1610), an operation of detecting a skin region (S1620), anoperation of detecting an ROI (S1630), and an operation of processing atleast two color channel values for the ROI (S1640), but the presentinvention is not limited thereto.

Also, the operation of acquiring the image (S1610), the operation ofdetecting the skin region (S1620), and the operation of detecting theROI (S1630) have been described above, and thus a redundant descriptionthereof will be omitted.

Also, the operation of processing the at least two color channel valuesfor the ROI (S1640) may include the above-described operations forprocessing the data for the ROI, and thus a redundant descriptionthereof will be omitted.

Also, the two color channels may be selected in consideration of theabsorbance of hemoglobin and the absorbance of oxyhemoglobin. Forexample, when the RGB color space is used, a red channel in which theabsorbance of hemoglobin is higher than the absorbance of oxyhemoglobinand a blue channel in which the absorbance of oxyhemoglobin is higherthan the absorbance of hemoglobin may be selected, but the presentinvention is not limited thereto.

Also, the oxygen saturation level measurement method 1600 according toan embodiment may include at least some of an operation of extractingtime-series data for the at least two color channel values for an imageframe group including at least some of the plurality of acquired imageframes (S1650) and acquiring an oxygen saturation level (S 1660), butthe present invention is not limited thereto.

Also, the operation of extracting the time-series data for the at leasttwo color channel values (S1650) may be performed on the image framegroup including the at least some of the plurality of acquired imageframes.

In this case, the image frame group may be a group of a plurality ofconsecutive or nonconsecutive image frames. For example, the image framegroup may refer to a group of consecutive image frames starting from afirst image frame up to a 180^(th) image frame. However, the presentinvention is not limited thereto, and the image frame group may refer toa group of at least some of the image frames starting from the firstimage frame up to the 180^(th) image frame.

Also, the operation of acquiring the oxygen saturation level (S1660) maybe performed on the image frame group including at least some of theplurality of acquired image frames.

In this case, an alternating current (AC) component and a direct current(DC) component of the at least two acquired pieces of time-series datamay be used to acquire the oxygen saturation level. In this case, the ACcomponent may refer to a difference between the maximum value and theminimum value of the time-series data and also may refer to a differencebetween the mean of maximum values and the mean of minimum values.However, the present invention is not limited thereto, and the ACcomponent may be understood as a typical AC component. Also, the DCcomponent may be understood as an average value of time-series data.However, the present invention is not limited thereto, and the DCcomponent may be understood as a typical DC component.

Also, an equation may be used to acquire the oxygen saturation level.

For example, when it is assumed that the absorbance of oxyhemoglobin inthe red channel is ε₁, that the absorbance of hemoglobin in the redchannel is ε₂, that the absorbance of oxyhemoglobin in the blue channelis ε₃, that the absorbance of hemoglobin in the blue channel is ε₄, andthat the proportion of oxyhemoglobin is S, Equation 2 may be obtained asfollows:

$\begin{matrix}{\frac{{S*ɛ_{3}} + {\left( {1 - S} \right)*ɛ_{4}}}{{S*ɛ_{1}} + {\left( {1 - S} \right)*ɛ_{2}}} = {\frac{A\; {C_{Blue}/D}\; C_{Blue}}{A\; {C_{Red}/D}\; C_{Red}}.}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

Here, the oxygen saturation level may be represented as S*100(%), butthe present invention is not limited thereto.

Also, it is obvious that various equations, other than theabove-described exemplary equation for obtaining an oxygen saturationlevel using at least two color channel values, are possible.

Also, the oxygen saturation level may be acquired based on one oxygensaturation level and may be acquired based on at least two oxygensaturation levels. For example, one oxygen saturation level may beacquired based on at least two pieces of time-series data, anotheroxygen saturation level may be acquired based on at least two otherpieces of time-series data. A final oxygen saturation level may beacquired based on at least two acquired oxygen saturation levels, butthe present invention is not limited thereto.

Also, the finally acquired oxygen saturation level and an oximeter maybe used to more accurately acquire the oxygen saturation level.

FIG. 11 is a flowchart illustrating an oxygen saturation levelmeasurement method according to another embodiment.

Referring to FIG. 11, an oxygen saturation level measurement method 1700according to an embodiment may include at least some of an operation ofacquiring an image for at least one of a plurality of acquired imageframes (S1710), an operation of detecting a skin region (S1720), anoperation of detecting an ROI (S1730), and an operation of processing atleast one color channel value for the ROI (S1740), but the presentinvention is not limited thereto.

Also, a plurality of infrared (IR) image frames may be acquired throughthe oxygen saturation level measurement method 1700 according to anembodiment. The oxygen saturation level measurement method 1700 mayinclude at least some of an operation of acquiring an IR image for atleast one of the acquired IR image frames (S1711), an operation ofdetecting a skin region (S1721), an operation of detecting an ROI(S1731), and an operation of processing IR data for the ROI (S1751), butthe present invention is not limited thereto.

Also, the operation of acquiring the image (S1710, S1711), the operationof detecting the skin region (S1720, S1721), and the operation ofdetecting the ROI (S1730, S1731) have been described above, and thus aredundant description thereof will be omitted.

Also, the operation of processing the at least one color channel valuefor the ROI (S1740) and the operation of processing the IR data for theROI (S1741) may include the above-described operations for processingthe data for the ROI, and thus a redundant description thereof will beomitted.

Also, the at least one color channel value and the wavelength range ofthe IR may be selected in consideration of the absorbance of hemoglobinand the absorbance of oxyhemoglobin. For example, when the RGB colorspace is used, a red channel in which the absorbance of hemoglobin ishigher than the absorbance of oxyhemoglobin may be selected, and an880-nm IR wavelength range in which the absorbance of oxyhemoglobin ishigher than the absorbance of hemoglobin may be selected, but thepresent invention is not limited thereto.

Also, the oxygen saturation level measurement method 1700 according toan embodiment may include at least some of an operation of extractingtime-series data for a color channel value for an image frame groupincluding at least some of the plurality of acquired image frames(S1750), an operation of extracting time-series data for the IR data(S1751), and an operation of acquiring an oxygen saturation level(S1760), but the present invention is not limited thereto.

Also, the above-described operations for extracting the time-series dataare applicable to the operation of extracting the time-series data forthe color channel value (S1750) and the operation of extracting thetime-series data for the IR data (S1751), and thus a redundantdescription thereof will be omitted.

Also, the operation of acquiring the oxygen saturation level (S1760) maybe performed on the image frame group including at least some of theplurality of acquired image frames and the IR image frame groupincluding at least some of the plurality of acquired IR image frames.

In this case, the image frame group including at least some of theplurality of acquired image frames may be identical to or different fromthe IR image frame group including at least some of the plurality ofacquired IR image frames.

For example, the image frame group and the IR image frame group may bedifferent from each other when the image frames and the IR image framesare acquired in different sequences, and the image frame group and theIR image frame group may be identical to each other when the imageframes and the IR image frames are acquired in the same sequence.

Also, an equation may be used to acquire the oxygen saturation level.

For example, when it is assumed that the absorbance of oxyhemoglobin inthe red channel is ε₁, that the absorbance of hemoglobin in the redchannel is ε₂, that the absorbance of oxyhemoglobin in a wavelength of880 nm is ε₃, that the absorbance of hemoglobin oxyhemoglobin in awavelength of 880 nm is ε₄, and that the proportion of oxyhemoglobin isS, Equation 3 may be obtained as follows:

$\begin{matrix}{\frac{{S*ɛ_{3}} + {\left( {1 - S} \right)*ɛ_{4}}}{{S*ɛ_{1}} + {\left( {1 - S} \right)*ɛ_{2}}} = {\frac{A\; {C_{880\mspace{14mu} n\; m}/D}\; C_{880\mspace{14mu} n\; m}}{A\; {C_{Red}/D}\; C_{Red}}.}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

Here, the oxygen saturation level may be represented as S*100(%), butthe present invention is not limited thereto.

Also, it is obvious that various equations other than theabove-described exemplary equation for obtaining an oxygen saturationlevel using at least one color channel value and IR data are available.

Also, the oxygen saturation level may be acquired based on one oxygensaturation level and may be acquired based on at least two oxygensaturation levels. For example, one oxygen saturation level may beacquired based on at least two pieces of time-series data, anotheroxygen saturation level may be acquired based on at least two otherpieces of time-series data. A final oxygen saturation level may beacquired based on at least two acquired oxygen saturation levels, butthe present invention is not limited thereto.

Also, the finally acquired oxygen saturation level and oximeter may beused to more accurately acquire the oxygen saturation level.

4.3 Various Embodiments of Blood Pressure Measurement Method

Blood pressure may refer to a pressure that blood flowing through ablood vessel exerts against the wall of the blood vessel.

Accordingly, the blood pressure may be affected by the speed of bloodflow, the thickness of the blood vessel wall, and waste productsaccumulated in blood vessels.

Also, when blood flows through a blood vessel, the volume of the bloodvessel may change, and the amount of blood contained in the blood vesselmay change.

Accordingly, when a rate of change in the volume of the blood vessel anda rate of change in the amount of blood are measured, the blood pressuremay be acquired.

For example, when a change in a blood vessel is measured at two pointsat different distances from a heart, a blood pressure may be obtainedbased on a difference between the changes at the two points in the bloodvessel, and a blood pressure may be obtained by extracting a featurethat can represent a change with time in the blood vessel, but thepresent invention is not limited thereto.

Also, it is obvious that, in addition to the above-described exemplaryprinciple, various principles for measuring blood pressures with lightare applicable.

FIG. 12 is a flowchart illustrating a blood pressure measurement methodaccording to an embodiment.

Referring to FIG. 12, a blood pressure measurement method 1800 accordingto an embodiment may include at least some of an operation of acquiringan image for at least one of a plurality of acquired image frames(S1810), an operation of detecting a skin region (S1820), an operationof detecting a first ROI (S1830), an operation of detecting a second ROI(S1831), an operation of processing data for the first ROI (S1840), andan operation of processing data for the second ROI (S1841), but thepresent invention is not limited thereto.

Also, the operation of acquiring the image (S1810) and the operation ofdetecting the skin region (S1820) have been described above, and thus aredundant description thereof will be omitted.

Also, the operation of detecting the first ROI (S1830) and the operationof detecting the second ROI (S1831) have been described in detail above,and thus a redundant description thereof will be omitted.

In this case, the first ROI and the second ROI may be set as two regionsat different distances from a subject's heart. For example, the firstROI may be set as an upper region of the subject's face, and the secondROI may be set as a lower region of the subject's face, but the presentinvention is not limited thereto. The first ROI may be set as a faceregion of the subject, and the second ROI may be set as a back-of-handregion of the subject.

Also, the operation of processing the data for the first ROI (S1840) andthe operation of processing the data for the second ROI (S1841) mayinclude the above-described operations for processing the data for theROI, and thus a redundant description thereof will be omitted.

Also, the blood pressure measurement method 1800 according to anembodiment may include at least some of an operation of extractingtime-series data for the first ROI and time-series data for the secondROI for an image frame group including at least some of the plurality ofacquired image frames (S1850), calculating a pulse transit time (PTT) onthe basis of the acquired time-series data (S1860), and acquiring ablood pressure (S1870), but the present invention is not limitedthereto.

Also, the above-described operations for extracting the time-series dataare applicable to the operation of extracting the time-series data forthe first ROI and the second ROI (S1850), and thus a redundantdescription thereof will be omitted.

Also, the operation of calculating the PTT on the basis of the acquiredtime-series data (S1860) may be performed on an image frame groupincluding at least some of the plurality of acquired image frames.

In this case, the PTT may be calculated based on local extremum valuesof the time-series data for the first ROI and the time-series data forthe second ROI. For example, the PTT may be calculated based on the timedifference between a local maximum value of the time-series data for thefirst ROI and a local maximum value of the time-series data for thesecond ROI. However, the present invention is not limited thereto, andthe PTT may be calculated based on the time difference between a localminimum value of the time-series data for the first ROI and a localminimum value of the time-series data for the second ROI.

Also, the PTT may be calculated based on inflection points of thetime-series data for the first ROI and the time series data for thesecond ROI. For example, the PTT may be calculated based on the timedifference between the inflection point of the time-series data for thefirst ROI and the inflection point of the time-series data for thesecond ROI, but the present invention is not limited thereto.

Also, the PTT may be calculated based on various points of thetime-series data for each ROI other than the local extremism values andthe inflection points.

Also, the time difference between the time-series data for the first ROIand the time-series data for the second ROI may be calculated based onframes acquired at points such as the local extremum values and theinflection points. For example, when a local maximum value is acquiredat a tenth frame in the time-series data for the first ROI and a localmaximum value is acquired at a twelfth frame in the time-series data forthe second ROI, the time difference between the time-series data for thefirst ROI and the time series data for the second ROI may be the timerequired to acquire two frames, and the PTT may be calculated based onthe time difference.

Also, the operation of acquiring the blood pressure (S1870) may beperformed on the image frame group including at least some of theplurality of acquired image frames.

Also, the PTT may be used to acquire the blood pressure. For example, afunction for the PTT may be used to acquire the blood pressure. Indetail, a function such as Equation 4 is available, and variousfunctions, such as a linear function, a quadratic function, alogarithmic function, and an exponential function, which have the PTT asa variable are available, but the present invention is not limitedthereto.

BP=f(PTT)   [Equation 4]

Also, the PTT and personal statistical data may be used to acquire theblood pressure. For example, a function for the PTT and a function forthe personal statistical data such as age, weight, and height may beused to acquire the blood pressure. In detail, Equation 5 below isavailable, and various functions, such as a linear function, a quadraticfunction, a logarithmic function, and an exponential function, whichhave the PTT, the weight, the height, and the age as variables areavailable, but the present invention is not limited thereto.

BP=af ₁(weight)+bf ₂(height)+cf ₃(age)+df ₄(PTT)   [Equation 5]

Also, a regression analysis method using the above-described functionmay be used to acquire the blood pressure, but the present invention isnot limited thereto.

Also, a machine learning method using the above-described function maybe used to acquire the blood pressure, but the present invention is notlimited thereto.

Also, it is obvious that various equations other than theabove-described exemplary equation for obtaining a blood pressure usinga PTT are available.

Also, the blood pressure may be acquired based on one blood pressure andmay be acquired based on at least two blood pressures. For example, oneblood pressure may be acquired based on a first PTT calculated based ontime-series data for the first ROI and the second ROI, and another bloodpressure may be acquired based on a second PTT calculated based on othertime-series data for the first ROI and the second ROI. A final bloodpressure may be acquired based on at least two acquired blood pressures,but the present invention is not limited thereto.

FIG. 13 is a flowchart illustrating a blood pressure measurement methodaccording to another embodiment.

Referring to FIG. 13, a blood pressure measurement method 1900 accordingto an embodiment may include at least some of an operation of acquiringan image for at least one of a plurality of acquired image frames(S1910), an operation of detecting a skin region (S1920), an operationof detecting an ROI (S1930), and an operation of processing data for theROI (S1940), but the present invention is not limited thereto.

In this case, the operation of acquiring the image (S1910), theoperation of detecting the skin region (S1920), the operation ofdetecting the ROI (S1930), and the operation of processing data for theROI (1940) have been described above, and thus a redundant descriptionthereof will be omitted.

Also, the blood pressure measurement method 1900 according to anembodiment may include at least some of an operation of extractingtime-series data for an image frame group including at least some of theplurality of acquired image frames (S1950), an operation of extracting afeature on the basis of the acquired time-series data (S1960), and anoperation of acquiring a blood pressure (S1970), but the presentinvention is not limited thereto.

Also, the above-described operations for extracting the time-series dataare applicable to the operation of extracting the time-series data(S1950), and thus a redundant description thereof will be omitted.

Also, the operation of extracting the feature on the basis of theacquired time-series data (S1960) may be performed on the image framegroup including the at least some of the plurality of acquired imageframes.

In this case, the feature may refer to a mathematical or physicalfeature of the acquired time-series data. For example, the feature mayrefer to a mathematical feature of the acquired time-series data such asa local maximum value, the mean of local maximum values, a local minimumvalue, the mean of local minimum values, a difference between a localmaximum value and a local minimum value, an average, an inflectionpoint, first-order differential data, second-order differential data,and a slope at a specific point or a physical feature of the acquiredtime-series data such as a blood variation, a blood change rate, a bloodvessel variation, and a blood vessel change rate, but the presentinvention is not limited thereto.

Also, it is obvious that the feature may include various features foracquiring a blood pressure in addition to the above-described exemplaryfeatures.

Also, the operation of acquiring the blood pressure (S1970) may beperformed on the image frame group including at least some of theplurality of acquired image frames.

Also, the feature may be used to acquire the blood pressure. Forexample, a function for the feature may be used to acquire the bloodpressure. In detail, a function such as Equation 6 is available, andvarious functions, such as a linear function, a quadratic function, alogarithmic function, and an exponential function, which have thefeature as a variable are available, but the present invention is notlimited thereto,

BP=f(feature)   [Equation 6]

Also, the feature and personal statistical data may be used to acquirethe blood pressure. For example, a function for the feature and afunction for the personal statistical data such as age, weight, andheight may be used to acquire the blood pressure. In detail, Equation 7below is available, and various functions, such as a linear function, aquadratic function, a logarithmic function, and an exponential function,which have the feature, the weight, the height, and the age as variablesare available, but the present invention is not limited thereto.

BP=af ₁(weight)+bf ₂(height)+cf ₃(age)+df ₄(feature)   [Equation 7]

Also, a regression analysis method using the above-described functionmay be used to acquire the blood pressure, but the present invention isnot limited thereto.

Also, a machine learning method using the above-described function maybe used to acquire the blood pressure, but the present invention is notlimited thereto.

Also, it is obvious that in addition to the above-described exemplaryequation, an equation for obtaining a blood pressure using a feature isavailable.

Also, the blood pressure may be acquired based on one blood pressure andmay be acquired based on at least two blood pressures. For example, oneblood pressure may be acquired based on a first feature calculated basedon time-series data, and another blood pressure may be acquired based ona second feature calculated based on other time-series data. A finalblood pressure may be acquired based on at least two acquired bloodpressures, but the present invention is not limited thereto.

4.4 Various Embodiments of Core Temperature Measurement Method

FIG. 14 is a flowchart illustrating a core temperature measurementmethod according to an embodiment.

Referring to FIG. 14, a core temperature measurement method 2000according to an embodiment may include at least some of an operation ofacquiring a skin temperature (S2010) and an operation of acquiring acore temperature (S2020), but the present invention is not limitedthereto.

Also, the operation of acquiring the skin temperature (S2010) may beperformed in a contactless manner.

For example, an image sensor such as a camera may be used to acquire theskin temperature. In detail, at least one color channel value of animage acquired from the image sensor such as a camera may be used toacquire the skin temperature. As an example, when the HSV color space isused, the skin temperature may be acquired using a saturation (S) value,but the present invention is not limited thereto.

Also, for example, a sensor such as a thermal imaging camera may be usedto acquire the skin temperature. An image sensor such as an infraredcamera is available, but the present invention is not limited thereto.

Also, the operation of acquiring the core temperature (S2020) may beperformed in a contactless manner.

For example, the skin temperature may be used to acquire the coretemperature. In detail, the core temperature may be acquired based onthe skin temperature using the relationship between the core temperatureand a portion where the skin temperature is measured.

In this case, the skin temperature may be acquired in the operation ofacquiring the skin temperature (S2010) and may also be acquired byanother external sensor.

Also, for example, an image sensor such as a camera may be used toacquire the core temperature. In detail, an image acquired by an imagesensor such as a camera may be used to acquire the skin temperature. Asan example, the core temperature may be acquired by applying the imageacquired by the image sensor such as a camera to a machine learningmodel for core temperature measurement as data, but the presentinvention is not limited thereto.

A method of acquiring a skin temperature and a method of acquiring acore temperature will be described in detail below.

First, a method of acquiring a skin temperature using an image sensorsuch as a camera will be described.

According to an embodiment, a brightness value of an acquired image maybe used to acquire a skin temperature. For example, a function for thebrightness value may be used to acquire the skin temperature. In detail,Equation 8 below is available, and various functions, such as a linearfunction, a quadratic function, a logarithmic function, and anexponential function, which have the brightness value as a variable areavailable, but the present invention is not limited thereto.

Skin Temperature=Af ₁(S)   [Equation 8]

Also, according to an embodiment, a brightness value of an acquiredimage and personal statistical data may be used to acquire a skintemperature. For example, a function for the brightness value and afunction for the personal statistical data such as age, race, and gendermay be used to acquire the skin temperature. In detail, Equation 9 belowis available, and various functions, such as a linear function, aquadratic function, a logarithmic function, and an exponential function,which have the brightness value, the age, the race, and the gender asvariables are available, but the present invention is not limitedthereto.

Skin Temperature=Af ₁(S)+Bf ₂(age)+Cf ₃(race)+Df ₄(gender)   [Equation9]

Also, according to an embodiment, a brightness value and a hue value ofan acquired image may be used to acquire a skin temperature. Forexample, a function for the brightness value and a function for the huevalue may be used to acquire the skin temperature. In detail. Equation10 below is available, and various functions, such as a linear function,a quadratic function, a logarithmic function, and an exponentialfunction, which have the brightness value and the hue value as variablesare available, but the present invention is not limited thereto.

Skin Temperature=Af ₁(S)+Bf ₂(H)   [Equation 10]

Also, according to an embodiment, a brightness value and a hue value ofan acquired image and personal statistical data may be used to acquire askin temperature. For example, a function for the brightness value, afunction for the hue value, and a function for the personal statisticaldata such as age, race, and gender may be used to acquire the skintemperature. In detail, Equation 11 below is available, and variousfunctions, such as a linear function, a quadratic function, alogarithmic function, and an exponential function, which have thebrightness value, the hue value, the age, the race, and the gender asvariables are available, but the present invention is not limitedthereto.

Skin Temperature=Af ₁(S)+Bf ₂(H)+Cf ₃(age)+Df ₄(race)+Ef ₅(gender)  [Equation 11]

Also, according to an embodiment, a brightness value, a hue value, and abrightness variation of an acquired image may be used to acquire a skintemperature. For example, a function for the brightness value, afunction for the hue value, and a function for the brightness variationmay be used to acquire the skin temperature. In detail, Equation 12below is available, and various functions, such as a linear function, aquadratic function, a logarithmic function, and an exponential function,which have the brightness value, the hue value, and the brightnessvariation as variables are available, but the present invention is notlimited thereto. Also, when such a variation is used, it is possible toreduce measurement noise caused by an external environment.

Skin Temperature=Af ₁(S)+Bf ₂(H)+Cf ₃(ΔS)   [Equation 12]

Also, according to an embodiment, a brightness value, a hue value, and abrightness variation of an acquired image and personal statistical datamay be used to acquire a skin temperature. For example, a function forthe brightness value, a function for the hue value, a function for thebrightness variation, and a function for the personal statistical datasuch as age, race, and gender may be used to acquire the skintemperature. In detail, Equation 13 below is available, and variousfunctions, such as a linear function, a quadratic function, alogarithmic function, and an exponential function, which have thebrightness value, the hue value, the brightness variation, the age, therace, and the gender as variables are available, but the presentinvention is not limited thereto.

Skin Temperature=Af ₁(S)+Bf ₂(H)+Cf ₃(ΔS)+Df ₄(age)+Ef ₅(race)+Ff₆(gender)   [Equation 13]

Also, according to an embodiment, a brightness value, a hue value, abrightness variation, and a hue variation of an acquired image may beused to acquire a skin temperature. For example, a function for thebrightness value, a function for the hue value, a function for thebrightness variation, and a function for the hue variation may be usedto acquire the skin temperature. In detail, Equation 14 below isavailable, and various functions, such as a linear function, a quadraticfunction, a logarithmic function, and an exponential function, whichhave the brightness value, the hue value, the brightness variation, andthe hue variation as variables are available, but the present inventionis not limited thereto. Also, when such a variation is used, it ispossible to reduce measurement noise caused by an external environment.

Skin Temperature=Af ₁(S)+Af ₂(H)+Cf ₃(ΔS)+Df ₄(ΔH)   [Equation 14]

Also, according to an embodiment, a brightness value, a hue value, abrightness variation, and a hue variation of an acquired image andpersonal statistical data may be used to acquire a skin temperature. Forexample, a function for the brightness value, a function for the huevalue, a function for the brightness variation, a function for the huevariation, and a function for the personal statistical data such as age,race, and gender may be used to acquire the skin temperature. In detail,Equation 15 below is available, and various functions, such as a linearfunction, a quadratic function, a logarithmic function, and anexponential function, which have the brightness value, the hue value,the brightness variation, the hue variation, the age, the race, and thegender as variables are available, but the present invention is notlimited thereto.

Skin Temperature=Af ₁(S)+Bf ₂(H)+Cf ₃(ΔS)+Df ₄(ΔH)+Ef ₅(age)+Ff₆(race)+Gf ₇(gender)   [Equation 15]

Also, a regression analysis method using the above-described functionmay be used to acquire the skin temperature, but the present inventionis not limited thereto.

Also, a deep learning method or a machine learning method using theabove-described function may be used to acquire the skin temperature,but the present invention is not limited thereto.

Also, it is obvious that various equations other than theabove-described exemplary equation are available. For example, anequation using at least some of the brightness value, the hue value, thebrightness variation, the hue variation, and the personal statisticaldata such as age, race, and gender is available, and also an equationusing other data is available.

Subsequently, a method of acquiring a core temperature using an imagesensor such as a camera will be described.

According to an embodiment, an acquired skin temperature may be used toacquire a core temperature. For example, a function for the skintemperature may be used to acquire the core temperature. In detail,Equation 16 below is available, and various functions, such as a linearfunction, a quadratic function, a logarithmic function, and anexponential function, which have the skin temperature as a variable areavailable, but the present invention is not limited thereto.

Care Temperature=Af(Skin Temp.)   [Equation 16]

According to an embodiment, when a core temperature is acquired indoors,the acquired skin temperature and indoor temperature may be used toacquire the core temperature. For example, a function for the skintemperature and a function for the indoor temperature may be used toacquire the core temperature. In detail, Equation 17 below is available,and various functions, such as a linear function, a quadratic function,a logarithmic function, and an exponential function, which have the skintemperature and the indoor temperature as variables are available, butthe present invention is not limited thereto.

Core Temperature=Af(Skin Temp.)+Bf(Room Temp.)   [Equation 17]

Also, according to an embodiment, when a core temperature is acquiredindoors, the acquired skin temperature and indoor temperature and aheart rate may be used to acquire the core temperature. For example, afunction for the skin temperature, a function for the indoortemperature, and a function for the heart rate may be used to acquirethe core temperature. In detail, Equation 18 below is available, andvarious functions, such as a linear function, a quadratic function, alogarithmic function, and an exponential function, which have the skintemperature, the indoor temperature, and the heart rate as variables areavailable, but the present invention is not limited thereto.

Core Temperature=Af(Skin Temp.)+Bf(Room Temp.)+Cf(Heartrate)   [Equation18]

Also, according to an embodiment, various skin temperatures may be usedat the same time. For example, when a core temperature is acquiredindoors, a first skin temperature acquired at a first body part, asecond skin temperature acquired at a second body part, an acquiredindoor temperature, and an acquired heart rate may be used to acquirethe core temperature. For example, a function for the first skintemperature, a function for the second skin temperature, a function forthe indoor temperature, and a function for the heart rate may be used toacquire the core temperature. In detail, Equation 19 below is available,and various functions, such as a linear function, a quadratic function,a logarithmic function, and an exponential function, which have thefirst skin temperature, the second skin temperature, the indoortemperature, and the heart rate as variables are available, but thepresent invention is not limited thereto.

Core Temperature=Af(Skin Temp.1)+Bf(Skin Temp.2)+Cf(RoomTemp.)+df(Heartrate)   [Equation 19]

Also, a regression analysis method using the above-described functionmay be used to acquire the core temperature, but the present inventionis not limited thereto.

Also, a deep learning method or a machine learning method using theabove-described. function may be used to acquire the core temperature,but the present invention is not limited thereto.

Also, it is obvious that various equations other than theabove-described exemplary equation are available. For example, anequation using at least some of the first skin temperature, the secondskin temperature, the indoor temperature, and the heart rate isavailable, and also an equation using personal statistical data such asage, gender, race, height, and weight is available.

5. Various Embodiments of Heart Rate Measurement Method

FIG. 15 is a flowchart illustrating a heart rate acquisition methodaccording to an embodiment.

Referring to FIG. 15, a heart rate acquisition method 2100 according toan embodiment may include at least some of an operation of acquiring animage (S2110), an operation of detecting a skin region (S2120), anoperation of detecting an ROI (S2130), an operation of processing datafor the ROI (S2140), an operation of acquiring a characteristic value(S2150), and an operation of acquiring a heart rate (S2160), but thepresent invention is not limited thereto.

In this case, the operation of acquiring the image (S2110), theoperation of detecting the skin region (S2120), and the operation ofdetecting the ROI (S2130) have been described above, and thus aredundant description thereof will be omitted.

Also, the operation of processing the data for the ROI (S2140) may beperformed on at least one of a plurality of acquired image frames.

Also, the operation of processing the data for the ROI (S2140) may beperformed to reduce noise caused by motion (motion artifact), noisecaused by external light, or the like.

Also, the operation of acquiring the characteristic value (S2150) may beperformed on an image frame group including at least some of theplurality of acquired image frames.

Also, the operation of acquiring the characteristic value (S2150) may beperformed to reduce noise caused by motion, noise caused by externallight, or the like.

Also, the operation of acquiring the heart rate (S2160) may be performedon an image frame group including at least some of the plurality ofacquired image frames.

In this case, the image frame group for acquiring the heart rate may beidentical to or different from the image frame group for acquiring thecharacteristic value. For example, the image frame group for acquiringthe characteristic value may include 18 image frames, and the imageframe group for acquiring the heart rate may include 180 image frames,but the present invention is not limited thereto.

The operation of processing the data for the ROI (S2140), the operationof acquiring the characteristic value (S2150), and the operation ofacquiring the heart rate (S2160) will be described in detail below.

FIG. 16 is a graph of color channel values according to an embodiment.

According to an embodiment, a color channel value for the ROI may beextracted to process the data for the ROI. In this case, the colorchannel value may be the mean of color channel values of pixels includedin the ROI and may be referred to as an average pixel value.

Referring to FIG. 16, a color channel value for the ROI corresponding tothe RGB color space may be extracted to process the data for the ROI. Indetail, a red channel value, which is the mean of red channel pixelvalues, a blue channel value, which is the mean of blue channel pixelvalues, and a green channel value, which is the mean of green channelpixel values, may be extracted.

For example, when color channel values corresponding to the RGB colorspace are extracted, a red channel pixel value, a green channel pixelvalue, and a blue channel pixel value of each pixel included in the ROImay be extracted. A red channel value, which is the mean of red channelpixel values included in the ROI, a blue channel value, which is themean of blue channel pixel values included in the ROI, and a greenchannel value, which is the mean of green channel pixel values includedin the ROI, may be extracted. However, the present invention is notlimited thereto, and color channel values corresponding to various colorspaces such as the HSV color space and the YCrCb color space may beextracted.

Also, a color channel value extracted according to a specific colorspace may be converted into another color space. For example, a colorchannel value extracted according to the RGB color space may beconverted into color channel values corresponding to various colorspaces such as the HSV color space and the YCrCb color space.

A color channel value extracted to process the data for the ROI may be acolor channel value obtained by weighting at least some of the colorchannel values extracted according to various color spaces and combiningthe color channel values.

Also, the color channel value may be extracted for each of a pluralityof image frames that are consecutively acquired or may be extracted fromat least some of the image frames.

The following description is based on a color channel value extractedaccording to the RGB color space. However, the present invention is notlimited thereto, and it is obvious that various color channel values areavailable.

FIG. 16A is a graph showing a red channel value extracted according tothe RGB color space, FIG. 16B is a graph showing a green channel valueextracted according to the RGB color space, and FIG. 16C is a graphshowing a blue channel value extracted according to the RCB color space.

As shown in FIG. 16, each color channel value may vary depending on theheartbeat.

However, each color channel value may vary due to subject movement or achange in intensity of external light while varying due to a heartbeat.

Accordingly, in order to acquire a heart rate using an extracted colorchannel value, there may be a need for an operation of reducing thevariation due to subject movement or the variation due to the change inintensity of external light and maximizing the variation due to theheartbeat.

5.1 Various Embodiments of Method of Processing Data for ROI

It is obvious that the above description is applicable to the method ofprocessing the data for the RGI, and thus a redundant descriptionthereof will be omitted.

FIG. 17 is a graph illustrating a noise reduction method according to anembodiment.

FIG. 17A is a graph showing a red channel value extracted according tothe RGB color space, and FIG. 17B is a graph showing a green channelvalue extracted according to the RGB color space.

Referring to FIGS. 17A and 17B, it can be seen that the extracted colorchannel values vary with time.

In this case, the extracted color channel values may vary depending on aheartbeat, subject movement, or a change in intensity of external light.

In detail, the variation of the color channel value being large and slowmay mean that the variation is more affected by subject movement or achange in intensity of external light, and the variation of the colorchannel value being small and quick may mean that the variation is moreaffected by a subject's heartbeat.

Therefore, the variation due to subject movement or the change inintensity of external light are greater than the variation due to theheartbeat, and thus a relative difference between at least two colorchannel values may be used to reduce the variation.

As an example, a difference between a green channel value and a redchannel value may be used to reduce noise. In detail, a green channelvalue and a red channel value acquired from the same image frame mayreflect the same motion and the same intensity of external light. Adifference between the green channel value and the red channel value ofthe same frame may reduce noise due to the subject's motion, the changein intensity of external light, and the like. However, the presentinvention is not limited thereto, and noise may be reduced using arelative difference between at least two color channel values.

FIG. 17C is a graph showing a difference between the green channel valueand the red channel value.

As shown in FIG. 17C, the difference between the green channel value andthe red channel value may reduce noise due to subject movement, thechange in intensity of external light, and the like.

Also, the above-described noise reduction method may be performed on atleast one of a plurality of acquired image frames or may be performed oneach of a plurality of consecutive image frames.

Also, although not shown in FIG. 17C, noise may be reduced using adifference between the green channel value and the blue channel value,and noise may be reduced using a difference between the red channelvalue and the blue channel value.

Also, in order to reduce noise using a relative difference between atleast two color channel values as described above, at least two colorchannel values may be selected to obtain the difference.

In this case, the at least two color channel values may be selected inconsideration of the absorbance of blood.

FIG. 18 is a diagram showing the absorbance of hemoglobin andoxyhemoglobin in a visible light range.

According to an embodiment, a red channel may be a channel including atleast a portion of a wavelength range from 620 nm to 750 nm, a greenchannel may be a channel including at least a portion of a wavelengthrange from 495 nm to 570 nm, and a blue channel may be a channelincluding at least a portion of a wavelength range from 450 nm to 495nm. However, the present invention is not limited thereto, and each ofthe red channel, the green channel, and the blue channel may be colorchannels that are generally understandable.

Referring to FIG. 18, the absorbance of hemoglobin and oxyhemoglobinaccording to the wavelength range of light can be seen. For example, asshown in FIG. 18, the absorbance of hemoglobin and oxyhemoglobin forlight in a wavelength range of 500 nm, which is included in the greenchannel, may be higher than the absorbance of hemoglobin andoxyhemoglobin for light in a wavelength range of 650 nm, which isincluded in the red channel.

Also, for example, as shown in FIG. 18, the absorbance of hemoglobin andoxyhemoglobin for light in a wavelength range of 550 nm, which isincluded in the green channel, may be higher than the absorbance ofhemoglobin and oxyhemoglobin for light in a wavelength range of 470 nm,which is included in the blue channel.

Also, when blood is transported throughout a body by a heartbeat, thevolume of a blood vessel or the amount of blood contained in a bloodvessel may change due to the flow of blood.

Accordingly, a color channel value including a wavelength range of lightthat is absorbed relatively more by hemoglobin and oxyhemoglobincontained in blood may vary relatively greatly due to a change in theamount of blood caused by a heartbeat.

On the contrary, a color channel value including a wavelength range oflight that is absorbed relatively less by hemoglobin and oxyhemoglobincontained in blood may vary relatively little due to a change in theamount of blood caused by a heartbeat.

Accordingly, according to an embodiment, at least two color channels maybe selected to reduce noise in consideration of the absorbance ofhemoglobin and oxyhemoglobin.

For example, according to an embodiment, a difference between a greenchannel value, which is absorbed relatively more by hemoglobin andoxyhemoglobin, and a red channel value, which is absorbed relativelyless by hemoglobin and oxyhemoglobin, may be used to reduce noise.

Also, for example, according to an embodiment, a difference between agreen channel value, which is absorbed relatively more by hemoglobin andoxyhemoglobin, and a blue channel value, which is absorbed relativelyless by hemoglobin and oxyhemoglobin, may be used to reduce noise.

Also, for example, according to an embodiment, a difference between ablue channel value, which is absorbed relatively more by hemoglobin andoxyhemoglobin, and a red channel value, which is absorbed relativelyless by hemoglobin and oxyhemoglobin, may be used to reduce noise.

Also, for example, according to an embodiment, the difference betweenthe green channel value and the red channel value and the differencebetween the green channel value and the blue channel value may be usedat the same time.

Also, the above examples have been described based on a difference, buta processed value obtained by applying a weight to each channel value,other than the difference, may be used to reduce noise caused by subjectmovement and noise caused by external light or the like.

For example, a processed value obtained by applying a weight to eachchannel value may be used as shown in Equation 20 below:

Processed Value=a*Red Channel Value+b*Blue Channel Value+c*Green ChannelValue.   [Equation 20]

Also, a, b, and c may be determined so that a+b+c=0 in order toefficiently remove noise caused by subject movement and noise caused byexternal light. In this case, each channel value may contain similarlevels of noise caused by motion and noise caused by external light inone image frame, and this can be advantageous in effectively reducingnoise.

5.2 Various Embodiments of Acquisition of Characteristic Value

A characteristic value may be acquired to reduce noise caused by subjectmovement and noise caused by the intensity of external light or thelike.

In this case, the characteristic value may be acquired for an imageframe group including at least some of a plurality of acquired imageframes.

Also, the characteristic value may be a value indicating a feature of anacquired color channel value or a processed value. For example, thecharacteristic value may refer to the mean, the deviation, the standarddeviation, and the like of color channel values or processed valuesincluded in an image frame group, but the present invention is notlimited thereto.

FIG. 19 is a diagram illustrating a characteristic value acquisitionmethod according to an embodiment.

FIG. 19A is a graph showing a color channel value acquired according toan embodiment, and more particularly, a graph showing a differencebetween a green channel value and a red channel value. However, thedifference between the green channel value and the red channel value isjust specified for convenience of description, and the present inventionis not limited thereto. Therefore, various color channel values,differences, processed values, and the like are available.

Referring to FIG. 19A, it can be seen that the difference between thegreen channel value and the red channel value (hereinafter referred toas a “G−R value”) may not change constantly over time.

In this case, the G−R value may not be constant due to subject movement.For example, the change in the G−R value may be small when the subjectmovement is small and may be large when the subject movement is large,but the present invention is not limited thereto.

Also, the G−R value may not be constant due to the intensity of externallight. For example, the change in the G−R value may be small when theintensity of external light is small and may be large when the intensityof external light is large, but the present invention is not limitedthereto.

Accordingly, a characteristic value may be extracted to reduce noisecaused by subject movement, the intensity of external light, or thelike.

Also, a window for the characteristic value may be set to extract thecharacteristic value.

In this case, the window for the characteristic value may refer to apredetermined time period and also may refer to the predetermined numberof frames. However, the present invention is not limited thereto, andthe window for the characteristic value may refer to a window forsetting a frame group including at least some of a plurality of framesin order to acquire the characteristic value.

FIG. 19B is a schematic diagram illustrating a window for acharacteristic value, and more particularly, a schematic diagramillustrating a window for a characteristic value by which 180 imageframes are set as ten equal parts, each of which includes 18 imageframes. However, for convenience of description, there is shown a windowfor a characteristic value by which 180 image frames are set as tenequal parts, each of which includes 18 image frames, but the presentinvention is not limited thereto, and the window for the characteristicvalue may be set in various ways and numbers.

Referring to FIG. 19B, a plurality of acquired image frames may begrouped by the window for the characteristic value. For example,referring to FIG. 19B, 180 image frames may be set as groups, each ofwhich includes 18 image frames, by the window for the characteristicvalue. In detail, 1^(st) to 18^(th) image frames may be included in afirst image frame group 2210, and 19^(th) to 36^(th) image frames may beincluded in a second image frame group 2220, but the present inventionis not limited thereto.

In this case, the characteristic value may be acquired for an imageframe group which is set by the window for the characteristic value. Forexample, the characteristic value may be acquired for color channelvalues for the first image frame group 2210 and may also be acquired forcolor channel values for the second image frame group 2220.

Also, for example, when the characteristic value is an average value,the mean of color channel values for an image frame group may beacquired. In detail, the mean of G−R values for the 1^(st) to 18^(th)image frames included in the first image frame group 2210 may beacquired, and the mean of G−R values for the 19^(th) to 36^(th) imageframes included in the second image frame group 2220 may he acquired,but the present invention is not limited thereto.

Also, for example, when the characteristic value is a standarddeviation, the standard deviation of color channel values for an imageframe group may be acquired. In detail, the standard deviation of G−Rvalues for the 1^(st) to 18^(th) image frames included in the firstimage frame group 2210 may be acquired, and the standard deviation ofG−R values for the 19^(th) to 36^(th) image frames included in thesecond image frame group 2220 may be acquired, but the present inventionis not limited thereto.

However, the present invention is not limited to the above examples, andvarious characteristic values may be acquired for an image frame group.

Also, the characteristic value may be acquired for at least some imageframes included in an image frame group obtained through division by thewindow for the characteristic value. For example, the characteristicvalue may be acquired for color channel values for at least some of the18 image frames included in the first image frame group 2210, and thecharacteristic value may be acquired for color channel values for atleast some of the 18 image frames included in the second image framegroup 2220.

Also, for example, when the characteristic value is a deviation, thedeviation of color channel values for at least some image framesincluded in an image frame group may be acquired. In detail, thedeviation of the G−R value of the first image frame included in thefirst image frame group 2210 from the mean of G−R values of the firstimage frame group 2210 may be acquired, and the deviation of the G−Rvalue of the ninth image frame included in the second image frame group2220 from the mean of G−R values of the second image frame group 2220may be acquired, but the present invention is not limited thereto.

Also, for example, when the characteristic value is a deviation, thedeviation of color channel values for at least some image framesincluded in an image frame group may be acquired. In detail, thedeviation of the G−R value of the first image frame included in thefirst image frame group 2210 from the mean of G−R values of the firstimage frame group 2210 may be acquired, and the deviation of the G−Rvalue of the second image frame included in the first image frame group2210 from the mean of G−R values of the first image frame group 2210 maybe acquired, but the present invention is not limited thereto.

Also, the acquired characteristic value may be normalized.

For example, when the characteristic value is a deviation, the deviationmay be normalized by the standard deviation. In detail, when thedeviation of the G−R value of the first image frame included in thefirst image frame group 2210 from the mean of G−R values of the firstimage frame group 2210 is acquired, the deviation of the G−R value ofthe first image frame may be normalized by the standard deviation of thefirst image frame group 2210. However, the present invention is notlimited thereto, and the deviation may be normalized in various ways.

Also, when the normalization is performed, the magnitude of variationmay be normalized. Thus, it is possible to better reflect a change dueto a heartbeat, and also it is possible to effectively reduce noisecaused by subject movement and noise caused by a change in the intensityof external light or the like.

FIG. 19C is a graph showing a characteristic value acquired according toan embodiment, and more particularly, a graph showing a deviationacquired based on a G−R value. However, the deviation acquired based onthe G−R value is just specified for convenience of description, and thepresent invention is not limited thereto. Therefore, variouscharacteristic values acquired based on various color channel values,differences, and processed values are available.

Referring to FIG. 19C, it can be seen that the variation is constantcompared to FIG. 19A.

Accordingly, by acquiring a characteristic value in the above-describedway, it is possible to reduce noise caused by subject movement and noisecaused by a change in the intensity of external light or the like, andalso it is possible to better reflect a change due to a heartbeat.

FIG. 20 is a diagram illustrating a characteristic value acquisitionmethod according to another embodiment.

FIG. 20A is a graph showing a color channel value acquired according toan embodiment, and more particularly, a graph showing a differencebetween a green channel value and a red channel value. However, thedifference between the green channel value and the red channel value isjust specified for convenience of description, and the present inventionis not limited thereto. Therefore, various color channel values,differences, processed values, and the like are available.

Referring to FIG. 20A, it can be seen that the difference between thegreen channel value and the red channel value (hereinafter referred toas a “G−R value”) may not change constantly over time.

In this case, the G−R value may not be constant due to subject movement.For example, the overall G−R value may be small in a time period 2301 inwhich a subject is positioned in a first state, and the overall G−Rvalue may be large in a time period 2302 in which a subject ispositioned in a second state different from the first state, but thepresent invention is not limited thereto.

Also, the G−R value may not be constant due to the intensity of externallight. For example, the intensity of external light in the time period2301 in which the subject is positioned in the first state may bedifferent from the intensity of external light in the time period 2302in which the subject is positioned in the second state. Thus, adifference may occur in the overall G−R value, but the present inventionis not limited thereto.

Accordingly, a characteristic value may be extracted to reduce noisecaused by subject movement, the intensity of external light, or thelike.

Also, a window for the characteristic value may be set to extract thecharacteristic value.

In this case, the window for the characteristic value may refer to apredetermined time period and also may refer to the predetermined numberof frames. However, the present invention is not limited thereto, andthe window for the characteristic value may refer to a window forsetting frame groups each including at least some of a plurality offrames in order to acquire the characteristic value.

Also, the frame groups that are set by the window may at least partiallyoverlap each other.

FIG. 20B is a schematic diagram illustrating a window for acharacteristic value, and more particularly, a schematic diagramillustrating a window for a characteristic value by which 180 imageframes are set as eight equal parts. However, for convenience ofdescription, there is shown a window for a characteristic value by which180 image frames are set as eight equal parts, but the present inventionis not limited thereto, and the window for the characteristic value maybe set in various ways and numbers.

Also, referring to FIG. 20B, a plurality of acquired image frames may begrouped by the window for the characteristic value. For example,referring to FIG. 20B, 180 image frames may be set as groups, each ofwhich includes 22 or 23 image frames, by the window for thecharacteristic value. In detail, 1^(st) to 22^(nd) image frames may beincluded in a first image frame group 2310.

Also, referring to FIG. 20B, the image frame groups set by the windowfor the characteristic value may at least partially overlap each other.For example, as shown in FIG. 20B, 1^(st) to 22^(nd) image frames may beincluded in a first image frame group 2310, 6^(th) to 28^(th) imageframes may be included in a second image frame group 2320, 12^(th) to33^(rd) image frames may be included in a third image frame group 2330,and 17^(th) to 39^(th) image frames may be included in a fourth imageframe group 2340, but the present invention is not limited thereto.

Also, referring to FIG. 20B, the image frame groups set by the windowfor the characteristic value may not overlap each other. For example, asshown in FIG. 20B, the 1^(st) to 22^(nd) image frames may be included inthe first image frame group 2310, and 23^(rd) to 45^(th) image framesmay be included in a fifth image frame group 2350, but the presentinvention is not limited thereto.

In this case, the characteristic value may be acquired for an imageframe group set by the window for the characteristic value or may beacquired for at least some image frames included in the image framegroup. However, the above description is applicable to this case, andthus a redundant description thereof will be omitted.

However, the processing of characteristic values acquired for the imageframes included in the at least partially overlapping image frame groupswill be described in detail below.

A plurality of characteristic values may be acquired for image framesincluded in a region where at least two image frame groups overlap eachother.

For example, at least two characteristic values may be acquired for the6^(th) to 22^(nd) image frames in a region where the first image framegroup 2310 and the second image frame group 2320 overlap each other.

In detail, a first deviation, which is the deviation of a G−R value ofthe sixth image frame from the mean of G−R values of the first imageframe group 2310, may be acquired, and a second deviation, which is thedeviation of a G−R value of the 6^(th) image frame from the mean of G−Rvalues of the second image frame group 2320, may be acquired, but thepresent invention is not limited thereto.

Also, a plurality of acquired characteristic value may be acquired asone characteristic value through a mathematical operation, For example,the deviation of the sixth image frame may be obtained by adding thefirst deviation and the second deviation, but the present invention isnot limited thereto.

Also, the above operations are applicable to acquire a characteristicvalue for an image frame included in a region where multiple, e.g.,three or four, image frame groups overlap each other.

Also, in addition to the above-described operation, a sliding windowscheme, which can be typically understood, is available.

FIG. 20C is a graph showing a characteristic value acquired according toan embodiment, and more particularly, a graph showing a deviationacquired based on a G−R value. However, the deviation acquired based onthe G−R value is just specified for convenience of description, and thepresent invention is not limited thereto. Therefore, variouscharacteristic values acquired based on various color channel values,differences, and processed values are available.

Referring to FIG. 20C, it can be seen that values are more uniform thanthose FIG. 19A as a whole.

In detail, the overall characteristic value in the time period 2301 inwhich a subject is positioned in the first state may become similar tothe overall characteristic value in the time period 2302 in which asubject is positioned in the second state.

Accordingly, by acquiring a characteristic value in the above-describedway, it is possible to reduce noise caused by subject movement and noisecaused by a change in the intensity of external light or the like, andalso it is possible to better reflect a change due to a heartbeat.

5.3 Various Embodiments of Method of Using Plurality of CharacteristicValues

A characteristic value acquired in the above-described methods may beinfluenced by a color channel value, a difference, and a processed valuewhich are to be used as a basis. Accordingly, by acquiring a pluralityof characteristic values on the basis of various color channel values,differences, and processed values and using the plurality ofcharacteristic values, it is possible to accurately acquire aphysiological parameter.

FIG. 21 is a diagram illustrating a method of using a plurality ofcharacteristic values.

FIG. 21A is a graph showing two characteristic values acquired accordingto an embodiment, and more particularly, a graph showing a firstcharacteristic value acquired based on a G−R value and a secondcharacteristic value acquired based on a G−B value. However, the firstcharacteristic value and the second characteristic value are justspecified for convenience of description, and the present invention isnot limited thereto. Therefore, characteristic values acquired based onvarious color channel values, differences, and processed values areavailable.

In this case, the first characteristic value acquired based on the G−Rvalue may be influenced by the G−R value. For example, when externallight is close to a blue channel, the G−R value may not reflect a changein blood due to a heartbeat well.

Alternatively, for example, the G−R value may be influenced by adifference between the absorbance of a green channel and the absorbanceof a red channel and thus may reflect a change in blood due to aheartbeat.

Also, the second characteristic value acquired based on the G−B valuemay be influenced by the G−B value. For example, when external light isclose to a red channel, the G−B value may not reflect a change in blooddue to a heartbeat well.

Alternatively, for example, the G−B value may be influenced by adifference between the absorbance of a green channel and the absorbanceof a blue channel and thus may reflect a change in blood due to aheartbeat.

Also, referring to FIG. 21A, the first characteristic value and thesecond characteristic value may have a complementary relationship. Forexample, the second characteristic value may reflect a change due to aheartbeat well in a period in which the first characteristic value doesnot reflect a change due to a heartbeat well, and vice versa.

Accordingly, the first characteristic value and the secondcharacteristic value may be used to reduce noise caused by a change inwavelength of external light or to reflect a change in blood due to aheartbeat better.

FIG. 21B is a graph showing a third characteristic value acquired usingthe first characteristic value and the second characteristic value, andmore particularly, a graph showing a third characteristic value acquiredby adding the first characteristic value and the second characteristicvalue. However, this is just specified for convenience of description,and the present invention is not limited thereto.

Also, the third characteristic value may be acquired based on amathematical operation between the first characteristic value and thesecond characteristic value. For example, the third characteristic valuemay be acquired based on an addition operation between the firstcharacteristic value and the second characteristic value. However, thepresent invention is not limited thereto, and the third characteristicvalue may be acquired based on various mathematical operations such as adifference operation and a multiplication operation.

Also, the third characteristic value may be acquired by assigningvarious weights to the first characteristic value and the secondcharacteristic value. For example, the third characteristic value may beacquired based on Equation 21 below, but the present invention is notlimited thereto.

Third Characteristic value=a*First Characteristic value+b*SecondCharacteristic value   [Equation 21]

Also, referring to FIGS. 21A and 21B, the third characteristic value mayreflect a change in blood due to a heartbeat better than the firstcharacteristic value and the second characteristic value and may reducenoise caused by a change in wavelength of external light.

5.4 Various Embodiments of Heart Rate Acquisition Method

In order to obtain a heart rate from data obtained by theabove-described methods, it may be necessary to detect a periodic changedue to a heartbeat. For example, in order to acquire a heart rate froman acquired characteristic value, there is a need to acquire awavelength or frequency component most often included in thecharacteristic value.

FIG. 22 is a graph showing a frequency component extracted from thegraph for the characteristic value shown in FIG. 21B. In detail, FIG. 22is a graph in which the graph for the characteristic value is FastFourier transformed into a frequency domain. However, the Fast Fouriertransform is just specified for convenience of description, and thepresent invention is not limited thereto. The graph for thecharacteristic value may be transformed according to the Fast Fouriertransform (FFT), discrete Fourier transform (DFT), short time Fouriertransform (STFT), or the like.

Also, the graph for the characteristic value may be transformed into afrequency domain as shown in FIG. 22.

In this case, a frequency index with the highest intensity may beacquired, and a heart rate may be acquired using Equation 22 below:

Heart Rate=Frequency Index*60.   [Equation 22]

For example, as shown in FIG. 22, when the frequency index with thehighest intensity is 1.2 Hz, the heart rate may be equal to 72 bpm.

Also, although not shown in FIG. 22, the graph for the characteristicvalue may be transformed into a frequency*measurement time domain.

In this case, a wavelength index with the highest intensity may beacquired, and a heart rate may be acquired using Equation 23 below:

Heart Rate Index/Measurement Time*60.   [Equation 23]

For example, although not shown in FIG. 22, when the wavelength indexwith the highest intensity is 8 and the measurement time is 6.6 seconds,the heart rate may be equal to 8/6.6*60, i.e., 72 bpm.

Also, various equations for acquiring a heart rate other than theabove-described exemplary equation are available.

Also, a preliminary heart rate may be acquired to acquire a heart rate.In this case, the preliminary heart rate may be a heart rate which iscalculated as a basis for acquiring the heart rate.

FIG. 23 is a diagram illustrating a heart rate acquisition methodaccording to an embodiment.

Prior to the description, the term “preliminary heart rate” used hereinmay refer to a heart rate acquired in the heart rate acquisition methodand may be a heart rate to be used as a basis for acquiring one heartrate. For example, at least two heart rates acquired in theabove-described heart rate acquisition method may be a first preliminaryheart rate and a second preliminary heart rate to be used as a basis foracquiring one final heart rate, but the present invention is not limitedthereto.

Also, a preliminary heart rate may itself be a final heart rate, and afinal heart rate may be acquired based on a plurality of preliminaryheart rates.

FIG. 23A is a graph showing a value acquired as time-series data. Forexample, the value shown in FIG. 23A may refer to a color channel valueacquired as time-series data, but the present invention is not limitedthereto. The value shown in FIG. 23A may refer to a difference or aprocessed value acquired as time-series data or may refer to acharacteristic value acquired as time-series data.

The above description is applicable to a method of transforming a graphshowing a value acquired as time-series data as shown in FIG. 23A into awavelength domain or a frequency domain, and thus a redundantdescription thereof will be omitted.

FIG. 23B is a schematic diagram illustrating a window for a preliminaryheart rate, and more particularly, a schematic diagram illustrating awindow for a preliminary heart rate with a period of six seconds.However, this is just specified for convenience of description, and thepresent invention is not limited thereto. The window may be set invarious sizes.

In this case, the window for the preliminary heart rate may refer to apredetermined time period and also may refer to the predetermined numberof frames. However, the present invention is not limited thereto, andthe window for the preliminary heart rate may refer to a window forsetting a frame group including at least some of a plurality of framesin order to acquire the preliminary heart rate.

Also, referring to FIG. 23B, a plurality of acquired image frames may begrouped by the window for the preliminary heart rate. For example, asshown in FIG. 20B, image frames acquired between zero and six secondsmay be included in a first image frame group 2410, but the presentinvention is not limited thereto.

Also, referring to FIG. 23B, the image frame groups set by the windowfor the preliminary heart rate may at least partially overlap eachother. For example, as shown in FIG. 23B, image frames acquired betweenzero and six seconds may be included in a first image frame group 2410,image frames acquired between 0.5 and 6.5 seconds may be included in asecond image frame group 2420, image frames acquired between one andseven seconds may be included in a third image frame group 2430, andimage frames acquired between 1.5 and 7.5 seconds may be included in afourth image frame group 2440, but the present invention is not limitedthereto.

In this case, the preliminary heart rate may be acquired for an imageframe group which is set by the window for the preliminary heart rate.For example, a first preliminary heart rate may be acquired based oncharacteristic values acquired from image frames included in the firstimage frame group 2410.

Also, a value acquired as time-series data from each image frame groupin order to acquire the preliminary heart rate may be transformed into awavelength domain or a frequency domain. For example, a value acquiredas time series data from the first image frame group may be transformedinto first frequency data 2460, a value acquired as time series datafrom the second image frame group may be transformed into secondfrequency data 2470, a value acquired as time series data from the thirdimage frame group may be transformed into third frequency data 2480, anda value acquired as time series data from the fourth image frame groupmay be transformed into fourth frequency data 2490, but the presentinvention is not limited thereto.

Also, the above-described heart rate acquisition method may be appliedto the acquisition of first to fourth preliminary heart rates from thefirst to fourth frequency data 2460, 2470, 2480, and 2490, and thus aredundant description thereof will be omitted.

Also, a heart rate may be acquired based on a plurality of preliminaryheart rates. For example, a heart rate may be acquired by performing amathematical operation on a plurality of preliminary heart rates, andmore particularly, may be acquired by performing a mathematicaloperation of extracting the mean, the maximum, the minimum, or the likeof the first to fourth preliminary heart rates, but the presentinvention is not limited thereto.

Also, a heart rate may be acquired based on a plurality of preliminaryheart rates. For example, a heart rate may be acquired by performing amathematical operation on some of a plurality of acquired preliminaryheart rates which have the same tens digit (excluding preliminary heartrates having different tens digits from the preliminary heart rates).

In detail, when it is assumed that the first preliminary heart rate is72 bpm, that the second preliminary heart rate is 80 bpm, that the thirdpreliminary heart rate is 85 bpm, and that the fourth preliminary heartrate is 73 bpm, a mathematical operation may be performed on the first,third, and fourth preliminary heart rates (excluding the secondpreliminary heart rate which has a different tens digit) to acquire aheart rate. In this case, the mathematical operation may be amathematical operation for extracting the mean, the maximum, theminimum, or the like.

Also, a heart rate may be acquired based on a plurality of preliminaryheart rates. For example, when four acquired preliminary heart rates arepaired in two pairs which have different tens digits from each other, amathematical operation may be performed on the pair of preliminary heartrates which have the same tens digit as a previously acquired heart rate(excluding the pair of preliminary heart rates which have different tensdigits) to acquire a heart rate.

In detail, when it is assumed that the first preliminary heart rate is72 bpm, that the second preliminary heart rate is 80 bpm, that the thirdpreliminary heart rate is 75 bpm, that the fourth preliminary heart rateis 73 bpm, and that a previously acquired heart rate is 75 bpm, amathematical operation may be performed on the first and fourthpreliminary heart rates (excluding the second and third preliminaryheart rates) to acquire a heart rate.

Also, when a heart rate is acquired using a plurality of preliminaryheart rates as described above, it is possible to increase robustnessagainst noise, and also it is possible to acquire an accurate heartrate.

5.5 Various Embodiments of Output of Heart Rate

A heart rate acquired by the above-described methods may be outputthrough a display or the like or may be transmitted to a terminal orserver using a communication unit. For convenience of description, suchan operation will be described below as an output of a heart rate.

When a heart rate is continuously measured in real time, an output heartrate may be corrected to provide stability and reliability to themeasured heart rate.

Also, when an output heart rate is corrected, it is possible to providestability and reliability to an acquired and output heart rate even if abad image is acquired from some of a plurality of acquired image frames.

FIG. 24 is a flowchart illustrating an output heart rate correctionmethod according to an embodiment.

Referring to FIG. 24, an output heart rate correction method 2500according to an embodiment may include an operation of acquiring a firstheart rate (S2510) and an operation of comparing a difference betweenthe first heart rate and a first-time-point heart rate to a referencevalue (S2520), but the present invention is not limited thereto.

The above-described heart rate acquisition methods may be applied to theoperation of acquiring the first heart rate (S2510), and thus aredundant description thereof will be omitted.

In the operation of comparing the difference between the first heartrate and the first-time-point heart rate to the reference value (S2520),the first-time-point heart rate may refer to a heart rate which isacquired or output before the first heart rate. For example, when thefirst heart rate is acquired at 6.5 seconds, the first-time-point heartrate may be a heart rate acquired at six seconds or a heart rate outputat six seconds, but the present invention is not limited thereto.

Also, the reference value may be determined as a certain numerical valueor a certain ratio. For example, the reference value may be set to ten.In this case, it may be determined whether the difference between thefirst heart rate and the first-time-point heart rate exceeds ten, butthe present invention is not limited thereto.

Also, when the difference between the first heart rate and thefirst-time-point heart rate is less than or equal to the referencevalue, an operation of outputting the first heart rate as asecond-time-point heart rate (S2531) may be performed. In this case, thesecond time point may be later than the first time point.

For example, when it is assumed that the first-time-point heart rate is72 bpm, that the first heart rate is 75 bpm, and that the referencevalue is ten, the heart rate output at the second time point may be 75bpm, but the present invention is not limited thereto.

Also, when the difference between the first heart rate and thefirst-time-point heart rate exceeds the reference value, an operation ofoutputting a heart rate obtained by correcting the first-time-pointheart rate as a second-time-point heart rate (S2532) may be performed.In this case, the second time point may be later than the first timepoint.

Also, in order to acquire a heart rate obtained by correcting thefirst-time-point heart rate, a mathematical operation may be performedon the first-time-point heart rate. For example, a mathematicaloperation such as addition or subtraction of a certain value to or fromthe first-time-point heart rate may be performed, but the presentinvention is not limited thereto.

For example, when it is assumed that the first-time-point heart rate is72 bpm, that the first heart rate is 85 bpm, and that the referencevalue is ten, the heart rate output at the second time point may be 75bpm, which is the first-time-point heart rate plus 3 bpm, but thepresent invention is not limited thereto.

Also, for example, when it is assumed that the first-time-point heartrate is 72 bpm, that the first heart rate is 61 bpm, and that thereference value is ten, the heart rate output at the second time pointmay be 69 bpm, which is the first-time-point heart rate minus 3 bpm, butthe present invention is not limited thereto.

5.6 Various Embodiments of Extraction of Heartbeat Signal

A heartbeat signal may refer to a signal varying depending on aheartbeat and also may refer to a signal to be estimated as varyingdepending on a heartbeat.

Also, a heartbeat signal may be extracted based on a plurality ofacquired image frames.

FIG. 25 is a diagram illustrating a heartbeat signal extraction methodaccording to an embodiment.

First, FIG. 25A is a graph showing a value acquired as time-series data.For example, the value shown in FIG. 25A may refer to a color channelvalue acquired as time-series data, but the present invention is notlimited thereto. The value shown in FIG. 23A may refer to a differenceor a processed value acquired as time-series data or may refer to acharacteristic value acquired as time-series data.

In this case, the value acquired as the time-series data may beextracted as a heartbeat signal through a band-pass filter. In detail,the value acquired as the time-series data may be extracted as aheartbeat signal through a band-pass filter for a frequency band orwavelength band corresponding to a heartbeat.

Also, the frequency band or wavelength band corresponding to theheartbeat may be a frequency band or wavelength band that can begenerally understood, but the present invention is not limited thereto.The frequency band or wavelength band may be a frequency band orwavelength band that is determined based on a heartbeat acquired by theabove-described methods.

For example, a typical heart rate may be 60 to 100 bpm, a correspondingfrequency band may be 1 to 1.67 Hz, and a corresponding band-pass filtermay be used. However, the present invention is not limited thereto.

Also, for example, when an acquired heart rate is 72 bpm, acorresponding frequency is 1.2 and a frequency band may be set based onthe corresponding frequency. In detail, when a frequency band of 0.5 Hzis set, the frequency band may be set to range from 0.95 Hz to 1.45 Hz,and a corresponding band-pass filter may be used, but the presentinvention is not limited thereto.

FIG. 25B is a heartbeat signal graph in which the value acquired astime-series data and shown in FIG. 25A is extracted as a heartbeatsignal through a band-pass filter.

Referring to FIG. 25B, it can be seen that a value acquired astime-series data may be extracted as a heartbeat signal through aband-pass filter.

5.7 Various Embodiments of Heart Rate Measurement Method Using InfraredLight

Basically, for a heart rate measurement method using infrared light, theabove-described heart rate measurement methods may be used.

FIG. 26 is a diagram illustrating a heart rate acquisition method usinginfrared light according to an embodiment.

In this case, near-infrared light in a wavelength band of 750 nm to 3000may be used as the infrared light. However, the present invention is notlimited thereto, and middle-infrared light, far-infrared, light, andextreme-infrared light may be used.

Referring to FIG. 26, a heart rate acquisition method 2600 usinginfrared light according to an embodiment may include at least some ofan operation of acquiring an image for at least one of a plurality ofacquired image frames (S2610), an operation of detecting a skin region(S2620), an operation of detecting an ROI (S2630), and an operation ofprocessing data for the ROI (S2640), but the present invention is notlimited thereto.

Also, the above description is applicable to the operation of acquiringthe image (S2610) and the operation of detecting the skin region(S2620), and thus a redundant description thereof will be omitted.

Also, the above-described methods of detecting the ROI are applicable todetails of the operation of detecting the ROI (S2630), and thus aredundant description thereof will be omitted.

Also, the operation of detecting the ROI (S2630) may include anoperation of detecting a first ROI, a second ROI, and a third ROI andmay be performed on at least one of the plurality of acquired imageframes.

Also, the first ROI, the second ROI, and the third ROI may at leastpartially overlap each other. For example, the first ROI may be set tobe included in the second ROI and the third ROI, and the second ROI maybe set to be included in the third ROI. However, the present inventionis not limited thereto, and the first to third ROIs may be set to atleast partially overlap each other.

Also, the first ROI, the second ROI, and the third ROI may be set not tooverlap each other. For example, the first ROI and the second ROI may bepositioned in an upper portion and a lower portion with respect to theleft cheek of a subject and the third ROI may be positioned on the rightcheek of the subject. However, the present invention is not limitedthereto, and the first to third ROIs may be set not to overlap eachother.

Also, the above-described methods of processing the data for the ROI areapplicable to details of the operation of processing the data for thefirst, second, and third ROIs (S2640), and thus a redundant descriptionthereof will be omitted.

Also, the operation of processing the data for the ROI (S2640) may beperformed on at least one of the plurality of acquired image frames.

Also, the operation of processing the data for the ROI (S2640) may beperformed on the first, second, and third ROIs.

Also, in order to process the data for the first to third ROIs, IRintensity values for the first to third ROIs may be extracted for atleast one of the plurality of acquired image frames. In this case, eachof the IR intensity values may be the mean of IR intensity values ofpixels included in the first, second, or third ROI and may be referredto as an average pixel value.

Also, when the above-described method of processing the data for the ROIis used, the IR intensity value for each ROI may correspond to theabove-described color channel value. For example, the IR intensity valueof the first ROI may correspond to a red channel value, the IR intensityvalue of the second ROI may correspond to a green channel value, and theIR intensity value of the third ROI may correspond to a blue channelvalue, but the present invention is not limited thereto.

Also, for example, the above-described G−R value may correspond to thedifference between the IR intensity value of the second ROI and the IRintensity value of the first ROI, and the above-described G−B value maycorrespond to the difference between the IR intensity value of thesecond ROI and the IR intensity value of the third ROI.

Accordingly, data may be processed based on the IR intensity value foreach ROI, and the details may follow the above-described data processingmethod for the ROI.

Also, the heart rate acquisition method 2600 using infrared lightaccording to an embodiment may include at least some of an operation ofextracting time-series data for an image frame group including at leastsome of a plurality of acquired image frames (S2650) and an operation ofacquiring a heart rate (S2660), but the present invention is not limitedthereto.

In this case, the above description is applicable to the operation ofextracting the time-series data (S2650) and the operation of acquiringthe heart rate (S2660), and thus a redundant description thereof will beomitted.

FIG. 27 is a diagram illustrating a heart rate acquisition method usinginfrared light according to an embodiment.

Referring to FIG. 27, at least two ROIs may be set in a face region of asubject. In detail, a first ROI 2710, a second ROI 2720, and a third ROI2730 may be set in the face region of the subject, but the presentinvention is not limited thereto.

Also, IR intensity values for the first to third ROIs 2710, 2720, and2730 may be extracted.

For example, the IR intensity value for the first ROI 2710 may beextracted (see FIG. 27A), the IR intensity value for the second ROI 2720may be extracted (see FIG. 27B), and the IR intensity value for thethird ROI 2730 may be extracted (see FIG. 27C), but the presentinvention is not limited thereto.

Also, data for the first to third ROIs 2710, 2720, and 2730 may beprocessed based on the IR intensity values extracted for the first tothird ROIs 2710, 2720, and 2730. However, details thereof have beendescribed above, and thus a redundant description thereof will beomitted.

Also, a characteristic value may be acquired based on the data processedfor the first to third ROIs 2710, 2720, and 2730. However, detailsthereof have been described above, and thus a redundant descriptionthereof will be omitted.

Also, a heart rate may be acquired using the characteristic valueacquired based on the IR intensity values extracted for the first tothird ROIs 2710, 2720, and 2730. However, details thereof have beendescribed above, and thus a redundant description thereof will beomitted.

6. Physiological Parameter Acquisition Method According to Embodiment

FIG. 28 is a flowchart illustrating a physiological parameteracquisition method according to an embodiment.

Referring to FIG. 28, the physiological parameter acquisition methodaccording to an embodiment may include at least some of an operation ofacquiring a plurality of image frames for a subject (S2810), anoperation of acquiring first, second, and third color channel values(S2820), an operation of calculating a first difference and a seconddifference (S2830), an operation of acquiring a first characteristicvalue and a second characteristic value (S2840), and an operation ofdetermining a physiological parameter of the subject on the basis of thefirst and second characteristic values (S2850).

In this case, a plurality of images may be acquired by a camera. Forexample, the plurality of images may be acquired by a camera such as avisible light camera and an IR camera.

Also, the plurality of images may be acquired from a camera placedoutside. For example, the plurality of images may be images acquiredfrom a camera such as a visible light camera or an IR camera placedoutside.

Also, the operation of acquiring the first, second, and third colorchannel values (S2820) may be performed on at least one of the pluralityof acquired image frames.

In this case, the first color channel value may refer to an averagepixel value for a first color channel of the image frame on which theacquisition operation is performed, the second color channel value mayrefer to an average pixel value for a second color channel of the imageframe, and the third color channel value may refer to an average pixelvalue for a third color channel of the image frame.

For example, the first color channel value may be a green channel value,the second color channel value may be a red channel value, and the thirdcolor channel value may be a blue channel value, but the presentinvention is not limited thereto.

Also, the operation of calculating the first difference and the seconddifference (S2830) may be performed on at least one of the plurality ofacquired image frames.

In this case, the first difference may refer to a difference between thefirst color channel value and the second color channel value. Forexample, the first difference may refer to a difference between thefirst color channel value and the second color channel value for thesame image frame, but the present invention is not limited thereto.

In detail, when the first color channel value is a green channel valueand the second color channel value is a red channel value, the firstdifference may be a G−R value, but the present invention is not limitedthereto.

Also, the second difference may refer to a difference between the firstcolor channel value and the third color channel value. For example, thesecond difference may refer to a difference between the first colorchannel value and the third color channel value for the same imageframe, but the present invention is not limited thereto.

In detail, when the first color channel value is a green channel valueand the second color channel value is a blue channel value, the seconddifference may be a G−B value, but the present invention is not limitedthereto.

Also, the operation of acquiring the characteristic value and the secondcharacteristic value (S2840) may be performed on an image frame groupincluding at least some of the plurality of acquired image frames.

For example, the first characteristic value may be acquired for a firstimage frame group, and the first image frame group may refer to an imageframe group acquired during a predetermined time.

Also, for example, the second characteristic value may be acquired for asecond image frame group, and the second image frame group may refer toan image frame group acquired during a predetermined time.

Also, the first characteristic value may be acquired based on the meanof first differences for the first image frame group and a firstdifference of an image frame included in the first image frame group.

For example, the first characteristic value may be a deviation acquiredbased on the mean of G−R values for the first image frame group and aG−R value for an image frame included in the first image frame group,but the present invention is not limited thereto.

Also, the second characteristic value may be acquired based on the meanof first differences for the second image frame group and a seconddifference of an image frame included in the second image frame group.

For example, the second characteristic value may be a deviation acquiredbased on the mean of G−B values for the second image frame group and aG−B value for an image frame included in the second image frame group,but the present invention is not limited thereto.

Here, the first image frame group may be identical to the second imageframe group. However, the present invention is not limited thereto, andthe first image frame group may be different from the second image framegroup.

Also, the first characteristic value may be the mean, standarddeviation, or the like of the first image frame group, but the presentinvention is not limited thereto.

Also, the first characteristic value may be the deviation or the like ofat least some image frames included in the first image frame group, butthe present invention is not limited thereto.

Also, the second characteristic value may be the mean, standarddeviation, or the like of the second image frame group, but the presentinvention is not limited thereto.

Also, the second characteristic value may be the deviation or the likeof at least some image frames included in the second image frame group,but the present invention is not limited thereto.

Also, the first characteristic value and the second characteristic valuemay be normalized. For example, the first characteristic value may benormalized using the standard deviation of the first image frame group,but the present invention is not limited thereto.

Also, for example, the second characteristic value may be normalizedusing the standard deviation of the second image frame group, but thepresent invention is not limited thereto.

Also, the operation of determining the physiological parameter of thesubject on the basis of the first and second characteristic values(S2850) may be performed on an image frame group including at least someof the plurality of acquired image frames.

In this case, the physiological parameter may be a heart rate, a bloodpressure, an oxygen saturation level, a core temperature, or the like,but the present invention is not limited thereto.

Also, the physiological parameter may be acquired based on the firstcharacteristic value and the second characteristic value.

For example, the physiological parameter may be acquired based on athird characteristic value obtained by adding the first characteristicvalue and the second characteristic value, but the present invention isnot limited thereto.

7. Various Embodiments of Method of Acquiring Plurality of PhysiologicalParameters and Plurality of Pieces of Physiological Information

FIG. 29 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters and a plurality of pieces of physiologicalinformation according to an embodiment.

Referring to FIG. 29, the method of acquiring a plurality ofphysiological parameters and a plurality of pieces of physiologicalinformation according to an embodiment may include acquiring a pluralityof physiological parameters on the basis of an acquired image 3000. Indetail, as shown in FIG. 29, a first physiological parameter 3010, asecond physiological parameter 3020, and a third physiological parameter3030 may be acquired based on the acquired image 3000. However, thepresent invention is not limited thereto, and at least two physiologicalparameters may be acquired. For example, four physiological parametersmay be acquired.

In this case, the acquired image 3000 may be a visible light image whichis acquired using a visible light camera or which is acquired from avisible light camera placed outside.

Also, the acquired image 3000 may be an infrared image which is acquiredusing an infrared camera or which is acquired from an infrared cameraplaced outside.

Also, the first to third physiological parameters 3010, 3020, and 30303may include at least one of a heart rate, an oxygen saturation level, ablood pressure, and a core temperature, but the present invention is notlimited thereto.

Also, the first to third physiological parameters 3010, 3020, and 3030may be acquired in association with each other, For example, the firstto third physiological parameters 3010, 3020, and 3030 may bephysiological parameters acquired in the same or similar states of thesubject, but the present invention is not limited thereto.

Also, the first to third physiological parameters 3010, 3020, and 3030may be acquired independently of each other. For example, the first tothird physiological parameters 3010, 3020, and 3030 may be physiologicalparameters acquired in different states of the subject, but the presentinvention is not limited thereto.

Also, the first to third physiological parameters 3010, 3020, and 3030may be output at the same time. For example, the first to thirdphysiological parameters 3010, 3020, and 3030 may be output at the sametime when measured in real time, but the present invention is notlimited thereto.

Also, the first to third physiological parameters 3010, 3020, and 3030may be output at different times. For example, the second physiologicalparameter may be output after the first physiological parameter isoutput, and the third physiological parameter may be output after thesecond physiological parameter is output.

Also, Referring to FIG. 29, the method of acquiring a plurality ofphysiological parameters and a plurality of pieces of physiologicalinformation according to an embodiment may include acquiring a pluralityof pieces of physiological information on the basis of the plurality ofacquired physiological parameters.

In detail, as shown in FIG. 29, first physiological information 3040,second physiological information 3050, and third physiologicalinformation 3060 may be acquired on the acquired first to thirdphysiological parameters 3010, 3020, and 3030, but the present inventionis not limited thereto.

In this case, the first to third physiological information 3040, 3050,and 3060 may include at least one of drowsiness information, stressinformation, excitement information, and emotion information, but thepresent invention is not limited thereto.

Also, the first to third physiological information 3040, 3050, and 3060may be acquired based on at least one of the first to thirdphysiological parameters 3010, 3020, and 3030, but the present inventionis not limited thereto.

Also, the first to third physiological information 3040, 3050, and 3060may be acquired in consideration of personal statistical data inaddition to the first to third physiological parameters 3010, 3020, and3030. In this case, the term “personal statistical data” may refer tocollectible personal statistical data of a subject such as age, gender,height, and weight, observable personal statistical data such as facialexpressions, wrinkles, and face color of a subject, or quantifiablepersonal statistical data such as statistical data (e.g., the averageblood pressure of people in their 20s, the average skin color of Asianpeople, the average height of men in their 30s, the average weight ofKorean men, etc.) calculated for a group including or relating to asubject.

Also, the first to third physiological information 3040, 3050, and 3060may be acquired based on independent physiological parameters. Forexample, the first to third physiological information 3040, 3050, and3060 may be acquired based on the first to third physiologicalparameters 3010, 3020, and 3030 acquired independently of each other.

Also, the first to third physiological information 3040, 3050, and 3060may be acquired based on associated physiological parameters. Forexample, the first to third physiological information 3040, 3050, and3060 may be acquired based on the first to third physiologicalparameters 3010, 3020, and 3030 acquired in association with each other.

Also, when the first to third physiological information 3040, 3050, and3060 are acquired based on associated physiological parameters, it ispossible to improve the accuracy of the physiological information. Forexample, when a heart rate and a blood pressure to be included in thefirst to third physiological parameters 3010, 3020, and 3030 are used toacquire hypertension information be included in the first to thirdphysiological information 3040, 3050, and 3060, the hypertensioninformation may be more accurate when the heart rate and the bloodpressure are associated with each other.

In detail, when it is assumed that a blood pressure is measured when asubject is exercising or excited, that a heart rate is measured when thesubject is stable, and that hypertension information is acquired basedon the blood pressure and the heart rate, the subject may be detected ashaving hypertension even if the subject is not hypertensive. However, onthe contrary, when it is assumed that a blood pressure and a heart rateare measured when a subject is exercising or excited and thathypertension information is acquired based on the blood pressure and theheart rate, the hypertension information may be accurately acquired.

The method of acquiring a plurality of physiological parameters and aplurality of pieces of physiological information will be described belowin detail.

FIG. 30 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 30, an image frame 3110 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the image frame 3110 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, the image frame 3110 may include a plurality of image frames, butthe present invention is not limited thereto.

Also, referring to FIG. 30, at least one pixel value 3120 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the pixel value 3120 may refer to an intensity value of atleast one pixel included in an image frame, and particularly, anintensity value of at least one pixel for at least one color channel.For example, when the image frame is a 640*480 image, the image framemay include 640*480 pixels, and a red channel pixel value, a greenchannel pixel value, and a blue channel pixel value for each pixel maybe acquired. However, the present invention is not limited thereto, andpixel values for color channels corresponding to various color spacesmay be acquired.

Also, the pixel value 3120 may be acquired only for at least some of aplurality of pixels included in the image frame.

Also, referring to FIG. 30, at least one color channel value 3130 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the color channel value 3130 may refer to the mean ofcolor channel pixel values. For example, the red channel value may referto the mean of red channel pixel values, the green channel value mayrefer to the mean of green channel pixel values, and the blue channelvalue may refer to the mean of blue channel pixel values, but thepresent invention is not limited thereto.

Although the color channel value and the processed value have beenseparately described throughout the specification, the processed valuemay be described below as a concept included in the color channel valuein order to describe a method of acquiring a plurality of physiologicalparameters.

Also, the color channel value 3130 may be described including aprocessed value. For example, the color channel value may be describedincluding a G−R value, which is a difference between the red channelvalue and the green channel value, but the present invention is notlimited thereto.

Also, the color channel value 3130 may be acquired for at least one ofthe plurality of acquired image frames.

Also, the color channel value 3130 may be acquired based on one colorchannel pixel value and may also be acquired based on a plurality ofcolor channel pixel values. For example, a red channel value may beacquired based on a red channel pixel value, and a G−R value may beacquired based on a green channel pixel value and a red channel pixelvalue, but the present invention is not limited thereto.

Also, referring to FIG. 30, at least one piece of time-series data maybe acquired to acquire a plurality of physiological parameters accordingto an embodiment.

In this case, time-series data 3140 may be acquired based on theacquired at least one color channel value 3130 and may also be acquiredfor an image frame group including at least some of the plurality ofacquired image frames.

In this case, when the time-series data 3140 is acquired based on atleast one color channel value 3130 acquired for a first image frame, thetime-series data may be acquired for an image frame group including thefirst image frame.

Also, the time-series data 3140 may be time-series data of the colorchannel value 3130. For example, the time-series data 3140 may includetime-series data of a red channel value, time-series data of a greenchannel value, time-series data of a blue channel value, time-seriesdata of a hue channel value, time-series data of a G−R value, ortime-series data of a G−B value, but the present invention is notlimited thereto.

Also, the time-series data 3140 may be time-series data of acharacteristic value acquired based on the color channel value 3130. Forexample, the time-series data may include the deviation, standarddeviation, or mean of color channel values for at least some imageframes included in an image frame group, but the present invention isnot limited thereto.

For example, the time-series data 3140 may be time-series data of a redchannel value and may also be time-series data of a G−R value, but thepresent invention is not limited thereto.

Also, for example, the time-series data 3140 may be time-series data forthe deviation of color channel values for at least some image framesincluded in an image frame group, but the present invention is notlimited thereto.

Also, the time-series data 3140 may be time-series data acquired basedon a plurality of pieces of time-series data. For example, thetime-series data 3140 may be first time-series data for a first imageframe group including a first image frame and may also be secondtime-series data for a second image frame group including a second imageframe, but the present invention is not limited thereto.

In detail, for example, when the time-series data 3140 is time-seriesdata for an image frame group including 180 image frames, thetime-series data 3140 may be acquired based on first time-series dataacquired for a first image frame group including 1^(st) to 18^(st) imageframes, second time-series data acquired for a second image frame groupincluding 19^(th) to 36^(th) image frames, third time-series dataacquired for a third image frame group including 37^(th) to 54^(th)image frames, fourth time-series data acquired for a fourth image framegroup including 55^(th) to 72^(nd) image frames, fifth image-series dataacquired for a fifth image frame group including 73^(rd) to 90^(th)image frames, sixth time-series data acquired for a sixth image framegroup including 91^(st) to 108^(th) image frames, seventh time-seriesdata acquired for a seventh image frame group including 109^(th) to126^(th) image frames, eighth time-series data acquired for an eighthimage frame group including 127^(th) to 144^(th) image frames, ninthtime-series data acquired for a ninth image frame group including145^(th) to 162^(nd) image frames, and tenth time-series data acquiredfor a tenth image frame group including 163^(rd) to 180^(th) imageframe.

Also, referring to FIG. 30, a feature 3150 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the feature 3150 may refer to a mathematical or physicalfeature of the acquired time-series data. For example, the feature mayrefer to a mathematical feature of the acquired time-series data such asa frequency component, a maximum value, the mean of maximum values, aminimum value, the mean of minimum values, a difference between amaximum value and a minimum value, a difference between the mean ofmaximum values and the mean of minimum values, an alternating current(AC) value, a direct current (DC) value, an average value, an inflectionpoint, first-order differential data, second-order differential data,and a slope at a specific point or a physical feature of the acquiredtime-series data such as a blood variation, a blood change rate, a bloodvessel variation, and a blood vessel change rate, but the presentinvention is not limited thereto.

Also, the feature 3150 may refer to a mathematical or physical featurebetween a plurality of pieces of time-series data. For example, thefeature may refer to a mathematical feature between the plurality ofacquired time-series data such as a time difference therebetween, a timedifference between local maximum values, a time difference between localminimum values, a time difference between inflection points, and thelike or a physical feature between the plurality of acquired time-seriesdata such as a pulse transit time (PTT), a difference between bloodchange rates, a time difference between blood vessel changes caused byblood, but the present invention is not limited thereto.

Also, the feature 3150 may be acquired for an image frame groupincluding at least some of the plurality of acquired image frames.

Also, the feature 3150 may be acquired based on at least one piece oftime-series data 3140. For example, the feature 3150 may be acquiredbased on one piece of time-series data and may also be acquired based onat least two pieces of time-series data, but the present invention isnot limited thereto.

Also, referring to FIG. 30, a physiological parameter 3160 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the physiological parameter 3160 may be a heart rate, anoxygen saturation level, a blood pressure, a core temperature, or thelike, but the present invention is not limited thereto.

Also, the physiological parameter 3160 may be acquired based ondifferent features. For example, a heart rate may be acquired based on afrequency component of time-series data, a blood pressure may beacquired based on a PTT value between two pieces of time-series data,and an oxygen saturation level may be acquired based on AC values and DCvalues of two pieces of time-series data, but the present invention isnot limited thereto.

Also, the physiological parameter 3160 may include a plurality ofphysiological parameters. For example, a heart rate, an oxygensaturation level, and a blood pressure may be acquired, but the presentinvention is not limited thereto.

Also, the physiological parameter 3160 may be acquired for an imageframe group including at least some of the plurality of acquired imageframes.

Also, the same image frame group may be used as a basis to acquire theplurality of physiological parameters 3160. For example, a heart rate,an oxygen saturation level, and a blood pressure may be acquired basedon an image frame group including 1^(st) to 180^(th) image frames, butthe present invention is not limited thereto.

Also, different image frame groups may be used as a basis to acquire theplurality of physiological parameters 3160. For example, a heart ratemay be acquired based on a first image frame group including 1^(st) to90^(th) image frames, an oxygen saturation level may be acquired basedon a second image frame group including 91^(st) to 180^(th) imageframes, and a blood pressure may be acquired based on a third imageframe including 181^(st) to 270^(th) image frames, but the presentinvention is not limited thereto.

Also, image frame groups used as a basis to acquire the plurality ofphysiological parameters 3160 may at least partially overlap each other.For example, a heart rate may be acquired based on a first image framegroup including 1^(st) to 180^(th) image frames, an oxygen saturationlevel may be acquired based on a second image frame group including30^(th) to 100^(th) image frames, and a blood pressure may be acquiredbased on a third image frame including 10^(th) to 180^(th) image frames,but the present invention is not limited thereto.

Also, image frames included in image frame groups used as a basis toacquire the plurality of physiological parameters 3160 may be the sameor different in number.

7.1 Method of Acquiring Plurality of Physiological Parameters Accordingto Embodiment

FIG. 31 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 31, an image frame 3210 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the image frame 3210 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, the image frame 3210 may include a plurality of image frames, butthe present invention is not limited thereto.

Also, referring to FIG. 31, at least one ROI 3220 may be set to acquirea plurality of physiological parameters according to an embodiment. Indetail, a first ROI, a second ROI, a third ROI, and a fourth ROI may beset.

In this case, the first ROI may be an ROI for acquiring an oxygensaturation level, the second ROI may be an ROI for acquiring a heartrate, the third ROI may be an ROI for acquiring a blood pressure, andthe fourth ROI may be an ROI for acquiring a core temperature, but thepresent invention is not limited thereto.

Also, the first to fourth ROIs may be the same or different from eachother.

Also, the first to fourth ROIs may at least partially overlap eachother.

Also, the sizes and areas of the first to fourth ROIs may be set basedon a physiological parameter to be acquired. For example, the second ROIfor acquiring a heart rate may be set to be large enough to include acheek region of a subject so as to detect a change in blood caused by aheartbeat well, and the third ROI for acquiring a blood pressure may beset to be vertically small and horizontally long enough to include acheek region of a subject so as to detect a fine blood flow rate well,but the present invention is not limited thereto.

Also, referring to FIG. 31, at least one pixel value 3230 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

Also, the pixel value 3230 may be acquired for at least one of theplurality of acquired image frames.

In detail, at least some of a red channel pixel value, a green channelpixel value, and a blue channel pixel value, which correspond to an RGBcolor space, may be acquired for the first ROI, at least some of a redchannel pixel value, a green channel pixel value, and a blue channelpixel value, which correspond to an RGB color space, may be acquired forthe second ROI, at least some of a red channel pixel value, a greenchannel pixel value, and a blue channel pixel value, which correspond toan RGB color space, may be acquired for the third ROI, and a hue channelpixel value, a saturation channel pixel value, and a value channel pixelvalue corresponding to an HSV color space may be acquired for the fourthROI, but the present invention is not limited thereto.

Also, referring to FIG. 31, at least one color channel value 3240 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the color channel value 3240 may refer to the mean ofcolor channel pixel values and also may refer to a processed value.However, this has been described in detail above, and thus a redundantdescription thereof will be omitted.

Also, the color channel value 3240 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two color channel values may be acquired for thefirst ROI for acquiring an oxygen saturation level.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe first ROI for acquiring an oxygen saturation level, but the presentinvention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the firstROI for acquiring an oxygen saturation level, but the present inventionis not limited thereto.

Also, the at least two color channel values may be selected inconsideration of the absorbance of hemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin ishigher than the absorbance of hemoglobin and a red channel in which theabsorbance of oxyhemoglobin is lower than the absorbance of hemoglobinmay be selected, and thus the at least two color channel values may beselected as a red channel value and a blue channel value, but thepresent invention is not limited thereto.

Also, at least two color channel values may be acquired for the secondROI for acquiring a heart rate.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe second ROI for acquiring a heart rate, but the present invention isnot limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the firstROI for acquiring a heart rate, but the present invention is not limitedthereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, at least two color channel values may be acquired for the thirdROI for acquiring a blood pressure.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe third ROI for acquiring a blood pressure, but the present inventionis not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the thirdROI for acquiring a blood pressure, but the present invention is notlimited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, at least two color channel values may be acquired for the fourthROI for acquiring a core temperature.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe fourth ROI for acquiring a core temperature, but the presentinvention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the fourthROI for acquiring a core temperature, but the present invention is notlimited thereto.

Also, the at least two color channel values may be selected inconsideration of a feature of a core temperature and a subject's skincolor.

For example, a saturation channel value associated with a subject's coretemperature and a hue channel value associated with a subject's skincolor may be selected, but the present invention is not limited thereto.

Also, referring to FIG. 31, at least one piece of time-series data 3250may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

Also, the time-series data 3250 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two pieces of time-series data may be acquired forthe first ROI for acquiring an oxygen saturation level. For example,first time-series data and second time-series data may be acquired forthe first ROI for acquiring an oxygen saturation level.

In this case, the first time-series data may be acquired based on acolor channel value acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI for acquiring an oxygensaturation level is a red channel value, the first time-series data maybe acquired based on the red channel value, but the present invention isnot limited thereto.

Also, the first time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the first time-series data is acquired based on a redchannel value acquired for a first image frame, the first time-seriesdata may be acquired for a first image frame group including the firstimage frame, but the present invention is not limited thereto.

Also, the second time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI for acquiring an oxygensaturation level is a blue channel value, the second time-series datamay be acquired based on the blue channel value, but the presentinvention is not limited thereto.

Also, the second time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the second time-series data is acquired based on ablue channel value acquired for a second image frame, the secondtime-series data may be acquired for a second image frame groupincluding the second image frame, but the present invention is notlimited thereto.

Also, the first and second image frame groups may be the same,different, or at least partially overlapping each other.

Also, at least one piece of time-series data may be acquired for thesecond ROI for acquiring a heart rate. For example, third time-seriesdata may be acquired for the second ROI for acquiring a heart rate.

In this case, the third time-series data may be acquired based on acolor channel value acquired for the second ROI. For example, when thecolor channel value acquired for the second ROI for acquiring a heartrate includes a G−R value and a G−B value, the third time-series datamay be acquired based on the G−R value and the G−B value, but thepresent invention is not limited thereto.

Also, the third time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the third time-series data is acquired based on a G−Rvalue and a G−B value acquired for a third image frame, the thirdtime-series data may be acquired for a third image frame group includingthe third image frame, but the present invention is not limited thereto.

Also, the third time-series data may be acquired based on acharacteristic value acquired for the second ROI. For example, the thirdtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the second ROI and acharacteristic value acquired based on the G−B value acquired for thesecond ROI, but the present invention is not limited thereto.

Also, at least one piece of time-series data may be acquired for thethird ROI for acquiring a blood pressure. For example, fourthtime-series data may be acquired for the third ROI for acquiring a bloodpressure.

In this case, the fourth time-series data may be acquired based on acolor channel value acquired for the third ROI. For example, when thecolor channel value acquired for the third ROI for acquiring a bloodpressure includes a G−R value and a G−B value, the fourth time-seriesdata may be acquired based on the G−R value and the G−B value, but thepresent invention is not limited thereto.

Also, the fourth time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the fourth time-series data is acquired based on a G−Rvalue and a G−B value acquired for a fourth image frame, the fourthtime-series data may be acquired for a fourth image frame groupincluding the fourth image frame, but the present invention is notlimited thereto.

Also, the fourth time-series data may be acquired based on acharacteristic value acquired for the third ROI. For example, the fourthtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the third ROI and acharacteristic value acquired based on the G−B value acquired for thethird ROI, but the present invention is not limited thereto.

Also, the first, second, third, and fourth image frame groups may be thesame. However, the present invention is not limited thereto, and theseimage frame groups may differ from each other or at least partiallyoverlap each other.

Also, referring to FIG. 31, at least one physiological parameteracquisition model 3251 may be used to acquire a plurality ofphysiological parameters according to an embodiment.

However, the physiological parameter acquisition model 3251 has beendescribed, and thus a redundant description thereof will be omitted.

Also, the physiological parameter acquisition model 3251 may be used toacquire a core temperature.

In detail, the physiological parameter acquisition model 3251 may have,as an input value, a color channel value acquired for the fourth ROI foracquiring a core temperature. For example, the physiological parameteracquisition model 3251 may have, as input values, a hue channel valueand a saturation channel value acquired for the fourth ROI, but thepresent invention is not limited thereto.

Also, the input value of the physiological parameter acquisition model3251 may include a color channel value, a characteristic value, the meanof color channel values for an image frame group, etc., but the presentinvention is not limited thereto.

Also, referring to FIG. 31, at least one feature 3260 may be acquired toacquire a plurality of physiological parameters according to anembodiment.

Also, the at least one feature 3260 may be acquired in consideration ofa physiological parameter to be acquired.

In detail, at least one feature may be acquired for the first ROI foracquiring an oxygen saturation level. For example, a first feature maybe acquired for the first ROI for acquiring an oxygen saturation level.

In this case, the first feature may be acquired based on a color channelvalue or time-series data acquired for the first ROI. For example, whenthe color channel value acquired for the first ROI includes a redchannel value and a blue channel value, the first feature may beacquired based on the red channel value and the blue channel value, butthe present invention is not limited thereto.

Also, for example, the first feature may be acquired based on firsttime-series data and second time-series data acquired for the first ROI,but the present invention is not limited thereto.

Also, the first feature may be a feature for acquiring an oxygensaturation level. For example, the first feature may include an AC valueand a DC value acquired based on the first time-series data and an ACvalue and a DC value acquired based on the second time-series data, butthe present invention is not limited thereto.

Also, for example, the first feature may include at least one of adifference between the mean of local maximum values and the mean of thelocal minimum values acquired based on the first time-series data, anaverage value acquired based on the first time-series data, a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the second time-series data, and anaverage value acquired based on the second time-series data, but thepresent invention is not limited thereto.

Also, the first feature may include a plurality of features. Forexample, the first feature may include at least two of a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the first time-series data, an averagevalue acquired based on the first time-series data, a difference betweenthe mean of local maximum values and the mean of the local minimumvalues acquired based on the second time-series data, and an averagevalue acquired based on the second time-series data, but the presentinvention is not limited thereto.

Also, at least one feature may be acquired for the second ROI foracquiring a heart rate. For example, a second feature may be acquiredfor the second ROI for acquiring a heart rate.

In this case, the second feature may be acquired based on a colorchannel value or time-series data acquired for the second ROI. Forexample, when the color channel value acquired for the second ROIincludes a G−R value and a G−B value, the second feature may be acquiredbased on the G−R value and the G−B value, but the present invention isnot limited thereto.

Also, for example, the second feature may be acquired based on thirdtime-series data acquired for the second ROI, but the present inventionis not limited thereto.

Also, the second feature may be a feature for acquiring a heart rate.For example, the second feature may include a frequency value, awavelength value, and a value for the number of cycle repetitions duringa measurement time, which are acquired based on the third time-seriesdata, but the present invention is not limited thereto.

Also, at least one feature may be acquired for the third ROI foracquiring a blood pressure. For example, a third feature may be acquiredfor the third ROI for acquiring a blood pressure.

In this case, the third feature may be acquired based on a color channelvalue or time-series data acquired for the third ROI. For example, whenthe color channel value acquired for the third ROI includes a G−R valueand a G−B value, the third feature may be acquired based on the G−Rvalue and the G−B value, but the present invention is not limitedthereto.

Also, for example, the third feature may be acquired based on fourthtime-series data acquired for the third ROI, but the present inventionis not limited thereto.

Also, the third feature may be a feature for acquiring a blood pressure.For example, the third feature may include a slope value, a maximumvalue, a minimum value, the mean of local maximum values, the mean oflocal minimum values, a difference between the mean of local maximumvalues and the mean of local minimum values, etc., which are acquiredbased on the fourth time-series data, but the present invention is notlimited thereto.

Also, the third feature may include a plurality of features. Forexample, the third feature may include at least two of a slope value, amaximum value, a minimum value, the mean of local maximum values, themean of local minimum values, a difference between the mean of localmaximum values and the mean of local minimum values, etc., which areacquired based on the fourth time-series data, but the present inventionis not limited thereto.

Also, at least one feature may be acquired for the fourth ROI foracquiring a core temperature. For example, a fourth feature may beacquired for the fourth ROI for acquiring a core temperature.

In this case, the fourth feature may be acquired based on an outputvalue of a physiological parameter acquisition model or a color channelvalue acquired for the fourth ROI. For example, when the color channelvalue acquired for the fourth ROI includes a hue channel value and asaturation channel value, the fourth feature may be acquired based onthe hue channel value and the saturation channel value, but the presentinvention is not limited thereto.

Also, for example, the fourth feature may be acquired based on theoutput value of the physiological parameter acquisition model, but thepresent invention is not limited thereto.

Also, the fourth feature may be a feature for acquiring a coretemperature. For example, the fourth feature may be a skin portion, askin temperature, etc., but the present invention is not limitedthereto.

Also, referring to FIG. 31, at least one physiological parameter 3270may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

In detail, each of the physiological parameters may be acquired based ona corresponding feature. For example, an oxygen saturation level may beacquired based on the first feature, a heart rate may be acquired basedon the second feature, a blood pressure may be acquired based on thethird. feature, and a core temperature may be acquired based on thefourth feature, but the present invention is not limited thereto.

In this case, the above equations may be used to acquire an oxygensaturation level on the basis of the first feature, and thus a redundantdescription thereof will be omitted.

Also, the above equations may be used to acquire a heart rate on thebasis of the second feature, and thus a redundant description thereofwill be omitted.

Also, the above equations may be used to acquire a blood pressure on thebasis of the third feature, and thus a redundant description thereofwill be omitted.

Also, the above equations may be used to acquire a core temperature onthe basis of the fourth feature, and thus a redundant descriptionthereof will be omitted.

Also, when the physiological parameter 3270 includes a plurality ofphysiological parameters, the physiological parameters may be acquiredat the same time. However, the present invention is not limited thereto,and the physiological parameters may be acquired at different times. Forexample, an oxygen saturation level and a heart rate may be acquired sixseconds after measurement, and a blood pressure may be acquired eightseconds after measurement, but the present invention is not limitedthereto.

Also, when the physiological parameter 3270 includes a plurality ofphysiological parameters, each of the physiological parameters may beacquired based on a corresponding image frame group. For example, anoxygen saturation level may be acquired based on a fifth image framegroup, a heart rate may be acquired based on a sixth image frame group,and a blood pressure may be acquired based. on a seventh image framegroup.

Also, the image frame groups for acquiring the physiological parameter3270 may be the same. However, the present invention is not limitedthereto, and these image frame groups may differ from each other or atleast partially overlap each other. For example, the fifth, sixth, andseventh image frame groups may be the same, different, or at leastpartially overlapped with each other.

Also, at least one preliminary physiological parameter may be acquiredto acquire the physiological parameter 3270. For example, at least fourpreliminary heart rates may be acquired to acquire a heart rate.

However, the above description is applicable to a method of acquiring aheart rate or a physiological parameter using a preliminary heart rateor a preliminary physiological parameter, and thus a redundantdescription thereof will be omitted.

Also, the number of preliminary physiological parameters for acquiringthe physiological parameter 3270 may be the same or different for eachphysiological parameter. For example, the number of preliminary heartrates for acquiring the heart rate may be at least four, and the numberof preliminary saturation levels for acquiring the oxygen saturationlevel and the number of preliminary blood pressures for acquiring theblood pressure may be at least two, but the present invention is notlimited thereto.

7.2 Method of Acquiring Plurality of Physiological Parameters Accordingto Embodiment

FIG. 32 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 32, an image frame 3310 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the image frame 3310 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, the image frame 3310 may include a plurality of image frames, butthe present invention is not limited thereto.

Also, referring to FIG. 32, at least one ROI 3320 may be set to acquirea plurality of physiological parameters according to an embodiment. Indetail, a first ROI, a second ROI, and a third ROI may be set.

In this case, the first ROI may be an ROI for acquiring an oxygensaturation level and a heart rate, and the second ROI and the third ROImay be ROIs for acquiring a blood pressure.

Also, the first to third ROIs may be the same or different from eachother.

Also, the first to third ROIs may at least partially overlap each other.

Also, the sizes and areas of the first to third ROIs may be set based ona physiological parameter to be acquired. For example, the size of thefirst ROI for acquiring a heart rate and an oxygen saturation level maybe set so that the first ROI includes a cheek region of a subject so asto detect a change in blood caused by a heartbeat well, and the areas ofthe second and third ROIs for acquiring a blood pressure may be setaccording to the direction of blood flow so as to reflect the bloodpressure well. However, the present invention is not limited thereto,and the first to third ROIs may be set in various sizes and variousareas.

Also, referring to FIG. 32, at least one pixel value 3330 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

Also, the pixel value 3330 may be acquired for at least one of theplurality of acquired image frames.

In detail, at least some of a red channel pixel value, a green channelpixel value, and a blue channel pixel value, which correspond to an RGBcolor space, may be acquired for the first ROI, at least some of a redchannel pixel value, a green channel pixel value, and a blue channelpixel value, which correspond to an RGB color space, may be acquired forthe second. ROI, and at least some of a red channel pixel value, a greenchannel pixel value, and a blue channel pixel value, which correspond toan RGB color space, may be acquired for the third ROI, but the presentinvention is not limited thereto.

Also, referring to FIG. 32, at least one color channel value 3340 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the color channel value 3340 may refer to the mean ofcolor channel pixel values and also may refer to a processed value.However, this has been described in detail above, and thus a redundantdescription thereof will be omitted.

Also, the color channel value 3340 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two color channel values may be acquired for thefirst ROI for acquiring an oxygen saturation level and a heart rate.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe first ROI, but the present invention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an value, which is a difference between a hue channel value and avalue channel value, and the like may be acquired for the first ROI, butthe present invention is not limited thereto.

Also, the at least two color channel values may be selected to acquirean oxygen saturation level in consideration of the absorbance ofhemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin ishigher than the absorbance of hemoglobin and a red channel in which theabsorbance of oxyhemoglobin is lower than the absorbance of hemoglobinmay be selected, and thus the at least two color channel values may beselected as a red channel value and a blue channel value, but thepresent invention is not limited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like inorder to acquire a heart rate.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, at least two color channel values may be acquired for the secondand third ROIs for acquiring a blood pressure.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe second and third ROIs for acquiring a blood pressure, but thepresent invention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the secondand third ROIs for acquiring a blood pressure, but the present inventionis not limited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, referring to FIG. 32, at least one piece of time-series data 3350may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

Also, the time-series data 3350 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two pieces of time-series data may be acquired forthe first ROI for acquiring an oxygen saturation level and a heart rate.For example, for the first ROI, first time-series data and secondtime-series data may be acquired to acquire an oxygen saturation level,and third time-series data may be acquired to acquire a heart rate.

In this case, the first time-series data may be acquired based on acolor channel value acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI is a red channel value,the first time-series data may be acquired based on the red channelvalue, but the present invention is not limited thereto.

Also, the first time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the first time-series data is acquired based on a redchannel value acquired for a first image frame, the first time-seriesdata may be acquired for a first image frame group including the firstimage frame, but the present invention is not limited thereto.

Also, the second time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI is a blue channel value, thesecond time-series data may be acquired based on the blue channel value,but the present invention is not limited thereto.

Also, the second time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the second time-series data is acquired based on ablue channel value acquired for a second image frame, the secondtime-series data may be acquired for a second image frame groupincluding the second image frame, but the present invention is notlimited thereto.

Also, the third time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI includes a G−R value and a G−Bvalue, the third time-series data may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, the third time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the third time-series data is acquired based on a G−Rvalue and a G−B value acquired for a third image frame, the thirdtime-series data may be acquired for a third image frame group includingthe third image frame, but the present invention is not limited thereto.

Also, the third time-series data may be acquired based on acharacteristic value acquired for the first ROI. For example, the thirdtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the first ROI and acharacteristic value acquired based on the G−B value acquired for thefirst ROI.

Also, the first, second, and third image frame groups may be the same.However, the present invention is not limited thereto, and these imageframe groups may differ from each other or at least partially overlapeach other.

Also, at least one piece of time-series data may be acquired for each ofthe second ROI and the third ROI to acquire a blood pressure. Forexample, fourth time-series data may be acquired for the second ROI, andfifth time-series data may be acquired for the third ROI, but thepresent invention is not limited thereto.

In this case, the fourth time-series data may be acquired based on acolor channel value acquired for the second ROI. For example, when thecolor channel value acquired for the second ROI includes a G−R value anda G−B value, the fourth time-series data may be acquired based on theG−R value and the G−B value, but the present invention is not limitedthereto.

Also, the fourth time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the fourth time-series data is acquired based on a G−Rvalue and a G−B value acquired for a fourth image frame, the fourthtime-series data may be acquired for a fourth image frame groupincluding the fourth image frame, but the present invention is notlimited thereto.

Also, the fourth time-series data may be acquired based on acharacteristic value acquired for the second ROI. For example, thefourth time-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the second ROI and acharacteristic value acquired based on the G−B value acquired for thesecond ROI, but the present invention is not limited thereto.

Also, the fifth time-series data may be acquired based on a colorchannel value acquired for the third ROI. For example, when the colorchannel value acquired for the third ROI includes a G−R value and a G−Bvalue, the fifth time-series data may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, the fifth time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the fifth time-series data is acquired based on a G−Rvalue and a G−B value acquired for a fifth image frame, the fifthtime-series data may be acquired for a fifth image frame group includingthe fifth image frame, but the present invention is not limited thereto.

Also, the fifth time-series data may be acquired based on acharacteristic value acquired for the third ROI. For example, the fifthtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the third ROI and acharacteristic value acquired based on the G−B value acquired for thethird ROI, but the present invention is not limited thereto.

Also, the first, second, third, fourth, and fifth image frame groups maybe the same. However, the present invention is not limited thereto, andthese image frame groups may differ from each other or at leastpartially overlap each other.

Also, referring to FIG. 32, at least one feature 3360 may be acquired toacquire a plurality of physiological parameters according to anembodiment.

Also, the at least one feature 3360 may be acquired in consideration ofa physiological parameter to be acquired.

In detail, at least one feature may be acquired for the first ROI foracquiring an oxygen saturation level and a heart rate. For example, afirst feature for the first ROI may be acquired to acquire an oxygensaturation level, and a second feature for the first ROI may be acquiredto acquire a heart rate.

In this case, the first feature may be acquired based on a color channelvalue or time-series data acquired for the first ROI. For example, whenthe color channel value acquired for the first ROI includes a redchannel value and a blue channel value, the first feature may beacquired based on the red channel value and the blue channel value, butthe present invention is not limited thereto.

Also, for example, the first feature may be acquired based on firsttime-series data and second time-series data acquired for the first ROI,but the present invention is not limited thereto.

Also, the first feature may be a feature for acquiring an oxygensaturation level. For example, the first feature may include an AC valueand a DC value acquired based on the first time-series data and an ACvalue and a DC value acquired based on the second time-series data, butthe present invention is not limited thereto.

Also, for example, the first feature may include at least one of adifference between the mean of local maximum values and the mean of thelocal minimum values acquired based on the first time-series data, anaverage value acquired based on the first time-series data, a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the second time-series data, and anaverage value acquired based on the second time-series data, but thepresent invention is not limited thereto.

Also, the first feature may include a plurality of features. Forexample, the first feature may include at least two of a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the first time-series data, an averagevalue acquired based on the first time-series data, a difference betweenthe mean of local maximum values and the mean of the local minimumvalues acquired based on the second time-series data, and an averagevalue acquired based on the second time-series data, but the presentinvention is not limited thereto.

Also, the second feature may be acquired based on a color channel valueor time-series data acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI includes a G−R value anda G−B value, the second feature may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, for example, the second feature may be acquired based on thirdtime-series data acquired for the first ROI, but the present inventionis not limited thereto.

Also, the second feature may be a feature for acquiring a heart rate.For example, the second feature may include a frequency value, awavelength value, and a value for the number of cycle repetitions duringa measurement time, which are acquired based on the third time-seriesdata, but the present invention is not limited thereto.

Also, at least one feature may be acquired for the second and third ROIsfor acquiring a blood pressure. For example, a third feature may beacquired for the second and third ROIs to acquire a blood pressure.

In this case, the third feature may be acquired based on color channelvalues or time-series data acquired for the second and third ROIs. Forexample, when the color channel values acquired for the second and thirdROIs include a G−R value and a G−B value, the third feature may beacquired based on the G−R value and the G−B value, but the presentinvention is not limited thereto.

Also, for example, the third feature may be acquired based on fourthtime-series data acquired for the second ROI and fifth time-series dataacquired for the third ROI, but the present invention is not limitedthereto.

Also, the third feature may be a feature for acquiring a blood pressure.For example, the third feature may include a slope value, a maximumvalue, a minimum value, the mean of local maximum values, the mean oflocal minimum values, and a difference between the mean of local maximumvalues and the mean of local minimum values, etc., which are acquiredbased on the fourth and fifth time-series data, but the presentinvention is not limited thereto.

Also, for example, the third feature may include a time differencebetween the fourth and fifth time-series data, a time difference betweenlocal maximum values, a time difference between local minimum values, atime difference between inflection points, etc. but the presentinvention is not limited thereto.

Also, the third feature may include a plurality of features. Forexample, the third feature may include at least two of theabove-described features, but the present invention is not limitedthereto.

Also, referring to FIG. 32, at least one physiological parameter 3370may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

In detail, each of the physiological parameters may be acquired based ona corresponding feature, For example, an oxygen saturation level may beacquired based on the first feature, a heart rate may be acquired basedon the second feature, and a blood pressure may be acquired based on thethird feature, but the present invention is not limited thereto.

In this case, the above equations may be used to acquire an oxygensaturation level on the basis of the first feature, and thus a redundantdescription thereof will be omitted.

Also, the above equations may be used to acquire a heart rate on thebasis of the second feature, and thus a redundant description thereofwill be omitted.

Also, the above equations may be used to acquire a blood pressure on thebasis of the third feature, and thus a redundant description thereofwill be omitted.

Also, when the physiological parameter 3370 includes a plurality ofphysiological parameters, the physiological parameters may be acquiredat the same time. However, the present invention is not limited thereto,and the physiological parameters may be acquired at different times. Forexample, an oxygen saturation level and a heart rate may be acquired sixseconds after measurement, and a blood pressure may be acquired eightseconds after measurement, but the present invention is not limitedthereto.

Also, when the physiological parameter 3370 includes a plurality ofphysiological parameters, each of the physiological parameters may beacquired based on a corresponding image frame group. For example, anoxygen saturation level may be acquired based on a sixth image framegroup, a heart rate may be acquired based on a seventh image framegroup, and a blood pressure may be acquired based on an eighth imageframe group.

Also, the image frame groups for acquiring the physiological parameter3370 may be the same. However, the present invention is not limitedthereto, and these image frame groups may differ from each other or atleast partially overlap each other. For example, the sixth, seventh, andeighth image frame groups may be the same, different, or at leastpartially overlapped with each other.

Also, at least one preliminary physiological parameter may be acquiredto acquire the physiological parameter 3370. For example, at least fourpreliminary heart rates may be acquired to acquire a heart rate,

However, the above description is applicable to a method of acquiring aheart rate or a physiological parameter using a preliminary heart rateor a preliminary physiological parameter, and thus a redundantdescription thereof will be omitted.

Also, the number of preliminary physiological parameters for acquiringthe physiological parameter 3370 may be the same or different for eachphysiological parameter. For example, the number of preliminary heartrates for acquiring the heart rate may be at least four, and the numberof preliminary saturation levels for acquiring the oxygen saturationlevel and the number of preliminary blood pressures for acquiring theblood pressure may be at least two, but the present invention is notlimited thereto.

7.3 Method of Acquiring Plurality of Physiological Parameters Accordingto Embodiment

FIG. 33 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 33, an image frame 3410 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the image frame 3410 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, the image frame 3410 may include a plurality of image frames, butthe present invention is not limited thereto.

Also, referring to FIG. 33, at least one ROI 3420 may be set to acquirea plurality of physiological parameters according to an embodiment. Indetail, a first ROI and a second ROI may be set.

In this case, the first ROI may be an ROI for acquiring an oxygensaturation level, a heart rate, and a blood pressure, and the first andsecond. ROIs may be ROIs for acquiring a blood pressure.

Also, the first and second ROIs may be the same or different from eachother.

Also, the first and second ROIs may at least partially overlap eachother.

Also, the sizes and areas of the first and second ROIs may be set basedon a physiological parameter to be acquired. For example, the size ofthe first ROI for acquiring a heart rate and an oxygen saturation levelmay be set so that the first ROI includes a cheek region of a subject soas to detect a change in blood caused by a heartbeat well, and the areasof the first and second ROIs for acquiring a blood pressure may be setaccording to the direction of blood flow so as to reflect the bloodpressure well. However, the present invention is not limited thereto,and the first and second ROIs may be set in various sizes and variousareas.

Also, referring to FIG. 33, at least one pixel value 3430 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

Also, the pixel value 3430 may be acquired for at least one of theplurality of acquired image frames.

In detail, at least some of a red channel pixel value, a green channelpixel value, and a blue channel pixel value, which correspond to an RGBcolor space, may be acquired for the first ROI, and at least some of ared channel pixel value, a green channel pixel value, and a blue channelpixel value, which correspond to an RGB color space, may be acquired forthe second ROI, but the present invention is not limited thereto.

Also, referring to FIG. 33, at least one color channel value 3440 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the color channel value 3440 may refer to the mean ofcolor channel pixel values and also may refer to a processed value.However, this has been described in detail above, and thus a redundantdescription thereof will be omitted.

Also, the color channel value 3440 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two color channel values may be acquired for thefirst ROI for acquiring an oxygen saturation level, a heart rate, and ablood pressure.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe first ROI, but the present invention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the firstROI, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to acquirean oxygen saturation level in consideration of the absorbance ofhemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin ishigher than the absorbance of hemoglobin and a red channel in which theabsorbance of oxyhemoglobin is lower than the absorbance of hemoglobinmay be selected, and thus the at least two color channel values may beselected as a red channel value and a blue channel value, but thepresent invention is not limited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like inorder to acquire a heart rate.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, at least two color channel values may be acquired for the firstand second ROIs for acquiring a blood pressure.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe first and second ROIs for acquiring a blood pressure, but thepresent invention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the firstand second ROIs for acquiring a blood pressure, but the presentinvention is not limited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which are differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, referring to FIG. 33, at least one piece of time-series data 3450may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

Also, the time-series data 3450 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two pieces of time-series data may be acquired forthe first ROI. For example, for the first ROI, first time-series dataand second time-series data may be acquired to acquire an oxygensaturation level, and third time-series data may be acquired to acquirea heart rate and a blood pressure.

In this case, the first time-series data may be acquired based on acolor channel value acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI is a red channel value,the first time-series data may be acquired based on a red channel value,but the present invention is not limited thereto.

Also, the first time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the first time-series data is acquired based on a redchannel value acquired for a first image frame, the first time-seriesdata may be acquired for a first image frame group including the firstimage frame, but the present invention is not limited thereto.

Also, the second time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI is a blue channel value, thesecond time-series data may be acquired based on the blue channel value,but the present invention is not limited thereto.

Also, the second time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the second time-series data is acquired based on ablue channel value acquired for a second image frame, the secondtime-series data may be acquired for a second image frame groupincluding the second image frame, but the present invention is notlimited thereto.

Also, the third time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI includes a G−R value and a G−Bvalue, the third time-series data may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, the third time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the third time-series data is acquired based on a G−Rvalue and a G−B value acquired for a third image frame, the thirdtime-series data may be acquired for a third image frame group includingthe third image frame, but the present invention is not limited thereto.

Also, the third time-series data may be acquired based on acharacteristic value acquired for the first ROI. For example, the thirdtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the first ROI and acharacteristic value acquired based on the G−B value acquired for thefirst ROI.

Also, at least one piece of time-series data may be acquired for thesecond ROI to acquire a blood pressure, and the blood pressure may beacquired based on the third time-series data acquired for the first ROIand fourth time-series data acquired for the second ROI.

In this case, the fourth time-series data may be acquired based on acolor channel value acquired for the second ROI. For example, when thecolor channel value acquired for the second ROI includes a G−R value anda G−B value, the fourth time-series data may be acquired based on theG−R value and the G−B value, but the present invention is not limitedthereto.

Also, the fourth time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the fourth time-series data is acquired based on a G−Rvalue and a G−B value acquired for a fourth image frame, the fourthtime-series data may be acquired for a fourth image frame groupincluding the fourth image frame, but the present invention is notlimited thereto.

Also, the fourth time-series data may be acquired based on acharacteristic value acquired for the second ROI. For example, thefourth time-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the second ROI and acharacteristic value acquired based on the G−B value acquired for thesecond ROI, but the present invention is not limited thereto.

Also, the first, second, third, and fourth image frame groups may be thesame. However, the present invention is not limited thereto, and theseimage frame groups may differ from each other or at least partiallyoverlap each other.

Also, referring to FIG. 33, at least one feature 3460 may be acquired toacquire a plurality of physiological parameters according to anembodiment.

Also, the at least one feature 3460 may be acquired in consideration ofa physiological parameter to be acquired.

In detail, at least one feature may be acquired for the first ROI. Forexample, a first feature for the first ROI may be acquired to acquire anoxygen saturation level, and a second feature for the first ROI may beacquired to acquire a heart rate.

In this case, the first feature may be acquired based on a color channelvalue or time-series data acquired for the first ROI. For example, whenthe color channel value acquired for the first ROI includes a redchannel value and a blue channel value, the first feature may beacquired based on the red channel value and the blue channel value, butthe present invention is not limited thereto.

Also, for example, the first feature may be acquired based on firsttime-series data and second time-series data acquired for the first ROI,but the present invention is not limited thereto.

Also, the first feature may be a feature for acquiring oxygen saturationlevel. For example, the first feature may include an AC value and a DCvalue acquired based on the first time-series data and an AC value and aDC value acquired based on the second time-series data, but the presentinvention is not limited thereto.

Also, for example, the first feature may include at least one of adifference between the mean of local maximum values and the mean of thelocal minimum values acquired based on the first time-series data, anaverage value acquired based on the first time-series data, a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the second time-series data, and anaverage value acquired based on the second time-series data, but thepresent invention is not limited thereto.

Also, the first feature may include a plurality of features. Forexample, the first feature may include at least two of a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the first time-series data, an averagevalue acquired based on the first time-series data, a difference betweenthe mean of local maximum values and the mean of the local minimumvalues acquired based on the second time-series data, and an averagevalue acquired based on the second time-series data, but the presentinvention is not limited thereto.

Also, the second feature may be acquired based on a color channel valueor time-series data acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI includes a G−R value anda G−B value, the second feature may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, for example, the second feature may be acquired based on thirdtime-series data acquired for the first ROI, but the present inventionis not limited thereto.

Also, the second feature may be a feature for acquiring a heart rate.For example, the second feature may include a frequency value, awavelength value, and a value for the number of cycle repetitions duringa measurement time, which are acquired based on the third time-seriesdata, but the present invention is not limited thereto.

Also, at least one feature may be acquired for the first and second ROIsfor acquiring a blood pressure. For example, a third feature may beacquired for the first and second ROIs to acquire a blood pressure.

In this case, the third feature may be acquired based on color channelvalues or time-series data acquired for the first and second ROIs. Forexample, when the color channel values acquired for the first and secondROIs include a G−R value and a G−B value, the third feature may beacquired based on the G−R value and the G−B value, but the presentinvention is not limited thereto.

Also, for example, the third feature may be acquired based on thirdtime-series data acquired for the first ROI and fourth time-series dataacquired for the second ROI, but the present invention is not limitedthereto.

Also, the third feature may be a feature for acquiring a blood pressure.For example, the third feature may include a slope value, a maximumvalue, a minimum value, the mean of local maximum values, the mean oflocal minimum values, a difference between the mean of local maximumvalues and the mean of local minimum values, etc., which are acquiredbased on the third and fourth time-series data, but the presentinvention is not limited thereto.

Also, for example, the third feature may include a time differencebetween the third and fourth time-series data, a time difference betweenlocal maximum values, a time difference between local minimum values, atime difference between inflection points, etc, but the presentinvention is not limited thereto.

Also, the third feature may include a plurality of features. Forexample, the third feature may include at least two of theabove-described features, but the present invention is not limitedthereto.

Also, referring to FIG. 33, at least one physiological parameter 3470may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

In detail, each of the physiological parameters may be acquired based ona corresponding feature. For example, an oxygen saturation level may beacquired based on the first feature, a heart rate may be acquired basedon the second feature, and a blood pressure may be acquired based on thethird feature, but the present invention is not limited thereto.

In this case, the above equations may be used to acquire an oxygensaturation level on the basis of the first feature, and thus a redundantdescription thereof will be omitted.

Also, the above equations may be used to acquire a heart rate on thebasis of the second feature, and thus a redundant description thereofwill be omitted.

Also, the above equations may be used to acquire a blood pressure on thebasis of the third feature, and thus a redundant description thereofwill be omitted.

Also, when the physiological parameter 3470 includes a plurality ofphysiological parameters. the physiological parameters may be acquiredat the same time. However, the present invention is not limited thereto,and the physiological parameters may be acquired at different times. Forexample, an oxygen saturation level and a heart rate may be acquired sixseconds after measurement, and a blood pressure may be acquired eightseconds after measurement, but the present invention is not limitedthereto.

Also, when the physiological parameter 3470 includes a plurality ofphysiological parameters, each of the physiological parameters may beacquired based on a corresponding image frame group. For example, anoxygen saturation level may be acquired based on a fifth image framegroup, a heart rate may be acquired based on a sixth image frame group,and a blood pressure may be acquired based on a seventh image framegroup.

Also, the image frame groups for acquiring the physiological parameter3470 may be the same. However, the present invention is not limitedthereto, and these image frame groups may differ from each other or atleast partially overlap each other. For example, the fifth, sixth, andseventh image frame groups may be the same, different, or at leastpartially overlapped with each other.

Also, at least one preliminary physiological parameter may be acquiredto acquire the physiological parameter 3470. For example, at least fourpreliminary heart rates may be acquired to acquire a heart rate.

However, the above description is applicable to a method of acquiring aheart rate or a physiological parameter using a preliminary heart rateor a preliminary physiological parameter, and thus a redundantdescription thereof will be omitted.

Also, the number of preliminary physiological parameters for acquiringthe physiological parameter 3470 may be the same or different for eachphysiological parameter. For example, the number of preliminary heartrates for acquiring the heart rate may be at least four, and the numberof preliminary saturation levels for acquiring the oxygen saturationlevel and the number of preliminary blood pressures for acquiring theblood pressure may be at least two, but the present invention is notlimited thereto.

7.4 Method of Acquiring Plurality of Physiological Parameters Accordingto Embodiment

FIG. 34 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 34, an image frame 3510 may be acquired to acquire aplurality of physiological parameters according to an embodiment.

In this case, the image frame 3510 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, the image frame 3510 may include a plurality of image frames, butthe present invention is not limited thereto.

Also, referring to FIG. 34, at least one ROI 3520 may be set to acquirea plurality of physiological parameters according to an embodiment. Indetail, a first ROI may be set.

In this case, the first ROI may be an ROI for acquiring an oxygensaturation level, a heart rate, and a blood pressure.

Also, the size and area of the first ROI may be set based on aphysiological parameter to be acquired. For example, the size of thefirst ROI for acquiring a heart rate and an oxygen saturation level maybe set so that the first ROI includes a cheek region of a subject so asto detect a change in blood caused by a heartbeat well. However, thepresent invention is not limited thereto, and the first ROI may be setin various sizes and various areas.

Also, referring to FIG. 34, at least one pixel value 3530 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

Also, the pixel value 3530 may be acquired for at least one of theplurality of acquired image frames.

In detail, at least some of a red channel pixel value, a green channelpixel value, and a blue channel pixel value, which correspond to an RGBcolor space, may be acquired for the first ROI, but the presentinvention is not limited thereto.

Also, referring to FIG. 34, at least one color channel value 3540 may beacquired to acquire a plurality of physiological parameters according toan embodiment.

In this case, the color channel value 3540 may refer to the mean ofcolor channel pixel values and also may refer to a processed value.However, this has been described in detail above, and thus a redundantdescription thereof will be omitted.

Also, the color channel value 3540 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least, two color channel values may be acquired for thefirst ROI for acquiring an oxygen saturation level, a heart rate, and ablood pressure.

For example, at least two of a red channel value, a green channel value,and a blue channel value, which correspond to an RGB color space, a huechannel value, a saturation channel value, and a value channel value,which correspond to an HSV color space, and the like may be acquired forthe first ROI, but the present invention is not limited thereto.

Also, for example, at least two of a G−R value, which is a differencebetween a red channel value and a green channel value, a G−B value,which is a difference between a green channel value and a blue channelvalue, an H−V value, which is a difference between a hue channel valueand a value channel value, and the like may be acquired for the firstROI, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to acquirean oxygen saturation level in consideration of the absorbance ofhemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin ishigher than the absorbance of hemoglobin and a red channel in which theabsorbance of oxyhemoglobin is lower than the absorbance of hemoglobinmay be selected, and thus the at least two color channel values may beselected as a red channel value and a blue channel value, but thepresent invention is not limited thereto.

Also, the at least two color channel values may be selected to reducenoise caused by motion and noise caused by external light or the like inorder to acquire a heart rate and a blood pressure.

For example, a G−R value, which is a difference between a green channelvalue absorbed relatively more by hemoglobin and oxyhemoglobin and a redchannel value absorbed relatively less by hemoglobin and oxyhemoglobin,and a G−B value, which is a difference between a green channel valueabsorbed relatively more by hemoglobin and oxyhemoglobin and a bluechannel value absorbed relatively less by hemoglobin and oxyhemoglobin,may be selected to reduce the noise, but the present invention is notlimited thereto.

Also, for example, a G−R value and a G−B value, which arc differencesbetween a green channel value reflecting relatively more of a changecaused by a heartbeat and red and blue channel values reflectingrelatively less of a change caused by a heartbeat, may be selected toreduce the noise, but the present invention is not limited thereto.

Also, referring to FIG. 34, at least one piece of time-series data 3550may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

Also, the time-series data 3550 may be acquired in consideration of aphysiological parameter to be acquired.

In detail, at least two pieces of time-series data may be acquired forthe first ROI. For example, for the first ROI, first time-series dataand second time-series data may be acquired to acquire an oxygensaturation level, and third time-series data may be acquired to acquirea heart rate and a blood pressure.

In this case, the first time-series data may be acquired based on acolor channel value acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI is a red channel value,the first time-series data may be acquired based on a red channel value,but the present invention is not limited thereto.

Also, the first time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the first time-series data is acquired based on a redchannel value acquired for a first image frame, the first time-seriesdata may be acquired for a first image frame group including the firstimage frame, but the present invention is not limited thereto.

Also, the second time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI is a blue channel value, thesecond time-series data may be acquired based on the blue channel value,but the present invention is not limited thereto.

Also, the second time-series data may be acquired for an image framegroup including at least sonic of a plurality of acquired image frames.

For example, when the second time-series data is acquired based on ablue channel value acquired for a second image frame, the secondtime-series data may be acquired for a second image frame groupincluding the second image frame, but the present invention is notlimited thereto.

Also, the third time-series data may be acquired based on a colorchannel value acquired for the first ROI. For example, when the colorchannel value acquired for the first ROI includes a G−R value and a G−Bvalue, the third time-series data may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, the third time-series data may be acquired for an image framegroup including at least some of a plurality of acquired image frames.

For example, when the third time-series data is acquired based on a G−Rvalue and a G−B value acquired for a third image frame, the thirdtime-series data may be acquired for a third image frame group includingthe third image frame, but the present invention is not limited thereto.

Also, the third time-series data may be acquired based on acharacteristic value acquired for the first ROI. For example, the thirdtime-series data may be acquired based on a characteristic valueacquired based on the G−R value acquired for the first ROI and acharacteristic value acquired based on the G−B value acquired for thefirst ROI.

Also, the first, second, and third image frame groups may be the same.However, the present invention is not limited thereto, and these imageframe groups may differ from each other or at least partially overlapeach other.

Also, referring to FIG. 34, at least one feature 3560 may be acquired toacquire a plurality of physiological parameters according to anembodiment.

Also, the at least one feature 3560 may be acquired in consideration ofa physiological parameter to be acquired.

In detail, at least one feature may be acquired for the first ROI. Forexample, a first feature for the first ROI may be acquired to acquire anoxygen saturation level, a second feature for the first ROI may beacquired to acquire a heart rate, and a third feature for the first ROImay be acquired to acquire a blood pressure, but the present inventionis not limited thereto.

In this case, the first feature may be acquired based on a color channelvalue or time-series data acquired for the first ROI. For example, whenthe color channel value acquired for the first ROI includes a redchannel value and a blue channel value, the first feature may beacquired based on the red channel value and the blue channel value, butthe present invention is not limited thereto.

Also, for example, the first feature may be acquired based on firsttime-series data and second time-series data acquired for the first ROI,but the present invention is not limited thereto.

Also, the first feature may be a feature for acquiring an oxygensaturation level, For example, the first feature may include an AC valueand a DC value acquired based on the first time-series data and an ACvalue and a DC value acquired based on the second time-series data, butthe present invention is not limited thereto.

Also, for example, the first feature may include at least one of adifference between the mean of local maximum values and the mean of thelocal minimum values acquired based on the first time-series data, anaverage value acquired based on the first time-series data, a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the second time-series data, and anaverage value acquired based on the second time-series data, but thepresent invention is not limited thereto.

Also, the first feature may include a plurality of features. Forexample, the first feature may include at least two of a differencebetween the mean of local maximum values and the mean of the localminimum values acquired based on the first time-series data, an averagevalue acquired based on the first time-series data, a difference betweenthe mean of local maximum values and the mean of the local minimumvalues acquired based on the second time-series data, and an averagevalue acquired based on the second time-series data, but the presentinvention is not limited thereto.

Also, the second feature may be acquired based on a color channel valueor time-series data acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI includes a G−R value anda G−B value, the second feature may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, for example, the second feature may be acquired based on thirdtime-series data acquired for the first ROI, but the present inventionis not limited thereto.

Also, the second feature may be a feature for acquiring a heart rate.For example, the second feature may include a frequency value, awavelength value, and a value for the number of cycle repetitions duringa measurement time, which are acquired based on the third time-seriesdata, but the present invention is not limited thereto.

Also, the third feature may be acquired based on a color channel valueor time-series data acquired for the first ROI. For example, when thecolor channel value acquired for the first ROI includes a G−R value anda G−B value, the third feature may be acquired based on the G−R valueand the G−B value, but the present invention is not limited thereto.

Also, for example, the third feature may be acquired based on thirdtime-series data acquired for the first ROI, but the present inventionis not limited thereto.

Also, the third feature may be a feature for acquiring a blood pressure.For example, the third feature may include a slope value, a maximumvalue, a minimum value, the mean of local maximum values, the mean oflocal minimum values, a difference between the mean of local maximumvalues and the mean of local minimum values, etc., which are acquiredbased on the third time-series data, but the present invention is notlimited thereto.

Also, the third feature may include a plurality of features. Forexample, the third feature may include at least two of a slope value, amaximum value, a minimum value, the mean of local maximum values, themean of local minimum values, a difference between the mean of localmaximum values and the mean of local minimum values, etc., which areacquired based on the third time-series data, but the present inventionis not limited thereto.

Also, referring to FIG. 34, at least one physiological parameter 3570may be acquired to acquire a plurality of physiological parametersaccording to an embodiment.

In detail, each of the physiological parameters may be acquired based ona corresponding feature. For example, an oxygen saturation level may beacquired based on the first feature, a heart rate may be acquired basedon the second feature, and a blood pressure may be acquired based on thethird feature, but the present invention is not limited thereto.

In this case, the above equations may be used to acquire an oxygensaturation level on the basis of the first feature, and thus a redundantdescription thereof will be omitted.

Also, the above equations may be used to acquire a heart rate on thebasis of the second feature, and thus a redundant description thereofwill be omitted.

Also, the above equations may be used to acquire a blood pressure on thebasis of the third feature, and thus a redundant description thereofwill be omitted.

Also, when the physiological parameter 3570 includes a plurality ofphysiological parameters, the physiological parameters may be acquiredat the same time. However, the present invention is not limited thereto,and the physiological parameters may be acquired at different times. Forexample, an oxygen saturation level and a heart rate may be acquired sixseconds after measurement, and a blood pressure may be acquired eightseconds after measurement, but the present invention is not limitedthereto.

Also, when the physiological parameter 3570 includes a plurality ofphysiological parameters, each of the physiological parameters may beacquired based on a corresponding image frame group. For example, anoxygen saturation level may be acquired based on a fourth image framegroup, a heart rate may be acquired based on a fifth image frame group,and a blood pressure may be acquired based on a sixth image frame group.

Also, the image frame groups for acquiring the physiological parameter3570 may be the same. However, the present invention is not limitedthereto, and these image frame groups may differ from each other or atleast partially overlap each other. For example, the fourth, fifth, andsixth image frame groups may be the same, different, or at leastpartially overlapped with each other.

Also, at least one preliminary physiological parameter may be acquiredto acquire the physiological parameter 3570. For example, at least fourpreliminary heart rates may be acquired to acquire a heart rate.

However, the above description is applicable to a method of acquiring aheart rate or a physiological parameter using a preliminary heart rateor a preliminary physiological parameter, and thus a redundantdescription thereof will be omitted.

Also, the number of preliminary physiological parameters for acquiringthe physiological parameter 3570 may be the same or different for eachphysiological parameter. For example, the number of preliminary heartrates for acquiring the heart rate may be at least four, and the numberof preliminary saturation levels for acquiring the oxygen saturationlevel and the number of preliminary blood pressures for acquiring theblood pressure may be at least two, but the present invention is notlimited thereto.

7.5 Method of Acquiring Plurality of Physiological Parameters Accordingto Embodiment

FIG. 35 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 35, a plurality of image frames 3600 may be acquiredto acquire a plurality of physiological parameters according to anembodiment.

In this case, the image frame 3600 may be an image frame acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, referring to FIG. 35, at least one physiological parameter may beacquired based on at least one feature.

For example, as shown in FIG. 35, a first physiological parameter 3660may be acquired based on a first feature 3610, a second physiologicalparameter 3670 may be acquired based on a second feature 3620, a thirdphysiological parameter 3680 may be acquired based on a third feature3630, and a fourth physiological parameter 3690 may be acquired based ona fourth feature 3640, but the present invention is not limited thereto.One physiological parameter may be acquired based on a plurality offeatures, and a plurality of physiological parameters may be acquiredbased on one feature.

In this case, the first to fourth features may include at least one ofan AC value, a DC value, a difference between the mean of local maximumvalues and the mean of local minimum values, an average value, afrequency component value, a wavelength component value, a value for thenumber of cycle repetitions during measurement, a slope value, a maximumvalue, a minimum value, the mean of local maximum values, the mean oflocal minimum values, an acquisition time difference, a time differencebetween local maximum values, a time difference between local minimumvalues, a time difference between inflection points, a skin temperature,and the like which are acquired based on at least one piece oftime-series data, but the present invention is not limited thereto.

Also, the first to fifth physiological parameters may include at leastone of a heart rate, an oxygen saturation level, a blood pressure, acore temperature, and a blood flow, but the present invention is notlimited thereto.

However, the features and physiological parameters have been describedin detail above, and thus a redundant description thereof will beomitted.

Also, the first to fourth features and/or the first to fourthphysiological parameters may be acquired based on an image frame groupincluding at least some of a plurality of image frames.

For example, the first feature and/or the first physiological parametermay be acquired based on a first image frame group, the second featureand/or the second physiological parameter may be acquired based on asecond image frame group, the third feature and/or the thirdphysiological parameter may be acquired based on a third image framegroup, and the fourth feature and/or the fourth physiological parametermay be acquired based on a fourth image frame group, but the presentinvention is not limited thereto.

In this case, each of the first to fourth image frames may include atleast two image frames.

For example, as shown in FIG. 35, the first image frame group mayinclude N image frames, the second image frame group may include M imageframes, the third image frame group may include K image frames, and thefourth image frame group may include L image frames, but the presentinvention is not limited thereto.

Also, the numbers of image frames included in the first to fourth imageframe groups may be the same.

For example, the first image frame group may include 180 image frames,the second image frame group may include 180 image frames, the thirdimage frame group may include 180 image frames, and the fourth imageframe group may include 180 image frames, but the present invention isnot limited thereto.

In this case, the first to fourth physiological parameters may beacquired or output at the same time. However, the present invention isnot limited thereto, and these physiological parameters may be acquiredor output at different times.

Also, in this case, the first to fourth physiological parameters may beacquired from the same condition of a subject and may be associated witheach other.

Also, the number of image frames included in the first to fourth imageframe groups may be different from each other.

For example, the first image frame group may include 180 image frames,the second image frame group may include 90 image frames, the thirdimage frame group may include 240 image frames, and the fourth imageframe group may include 360 image frames, but the present invention isnot limited thereto.

In this case, the first to fourth physiological parameters may beacquired or output at different times. However, the present invention isnot limited thereto, and these physiological parameters may be acquiredor output at the same time.

Also, the numbers of image frames included in the first to fourth imageframe groups may differ depending on a physiological parameter to beacquired.

For example, when it is assumed that the first physiological parameteris a heart rate, that the second physiological parameter is an oxygensaturation level, that the third physiological parameter is a bloodpressure, and that the fourth physiological parameter is a coretemperature, the first image frame group may include 180 image frames,the second image frame group may include 120 image frames, the thirdimage frame group may include 60 image frames, and the fourth imageframe group may include 60 image frames, but the present invention isnot limited thereto. The number of image frames included in an imageframe group may be set in consideration of a physiological parameter tobe acquired.

FIG. 36 is a diagram illustrating a method of acquiring a plurality ofassociated physiological parameters according to an embodiment.

Referring to FIG. 36, a plurality of image frames 3700 may be acquiredto acquire a plurality of associated physiological parameters accordingto an embodiment.

In this case, the image frames 3700 may be image frames acquired from avisible light image, an infrared image, etc., but the present inventionis not limited thereto.

Also, referring to FIG. 36, at least one physiological parameter may beacquired based on at least one image frame group.

In detail, as shown in FIG. 36, a first physiological parameter 3760 maybe acquired based on a first image frame group 3710, a secondphysiological parameter 3770 may be acquired based on a second imageframe group 3720, a third physiological parameter 3780 may be acquiredbased on a third image frame group 3730, and a fourth physiologicalparameter 3790 may be acquired based on a fourth image frame group 3740.

In this case, the above description is applicable to each image framegroup acid each physiological parameter, and thus a redundantdescription thereof will be omitted.

Referring to FIG. 36, the first physiological parameter 3760, the secondphysiological parameter 3770, the third physiological parameter 3780,and the fourth physiological parameter 3790 may at least partiallyoverlap each other to acquire the first image frame group 3710, thesecond image frame group 3720, the third image frame group 3730, and thefourth image frame group 3740 in association with each other.

For example, as shown in FIG. 36, the first image frame group 3710 mayinclude 1^(st) to 13^(th) image frames, the second image frame group3720 may include 6^(th) to 19^(th) image frames, the third image framegroup 3730 may include 4^(th) to 17^(th) image frames, and the fourthimage frame group 3740 may include 9^(th) to 23^(rd) image frames, butthe present invention is not limited thereto.

Accordingly, the first to fourth image frame groups 3710, 3720, 3730,and 3740 may include 9^(th) to 13^(th) image frames in common.

Also, as described above, the first to fourth image frame groups atleast partially overlap each other, and thus the first to fourthphysiological parameters may be acquired in association with each other.

For example, since the first to fourth image frame groups include the9^(th) to 13^(th) image frames in common, the first to fourthphysiological parameters may be acquired in the same state of a subject,and thus the first to fourth physiological parameters may be acquired inassociation with each other.

In detail, for example, when the state of the subject at a time at whichthe 9^(th) image frame is acquired is a first state, the first to fourthphysiological parameters may be acquired by reflecting the first stateof the subject in common, and thus the first to fourth physiologicalparameters may be acquired in association with each other.

Also, when the first to fourth physiological parameters 3760, 3770,3780, and 3790 are acquired in association with each other, it ispossible to accurately obtain the physiological parameters of thesubject.

Also, at least one piece of physiological information may be acquiredbased on the first to fourth physiological parameters 3760, 3770, 3780,and 3790.

Also, when the physiological information is acquired based on associatedphysiological parameters, it is possible to improve the accuracy of thephysiological information. For example, when a heart rate and a bloodpressure to be included in the first to fourth physiological parametersare used to acquire hypertension information to be included in thephysiological information, the hypertension information may be moreaccurate when the heart rate and the blood pressure are associated witheach other.

In detail, when it is assumed that a blood pressure is measured when asubject is exercising or excited, that a heart rate is measured when thesubject is stable, and that hypertension information is acquired basedon the blood pressure and the heart rate, the subject may be detected ashaving hypertension even if the subject is not hypertensive. However, onthe contrary, when it is assumed that a blood pressure and a heart rateare measured when a subject is exercising or excited and thathypertension information is acquired based on the blood pressure and theheart rate, the hypertension information may be accurately acquired.

FIG. 37 is a diagram illustrating a method of acquiring a plurality ofphysiological parameters according to an embodiment.

Referring to FIG. 37, at least one physiological parameter may beacquired based on at least one preliminary physiological parameter.

For example, as shown in FIG. 37, a first physiological parameter may beacquired based on N first preliminary physiological parameters, a secondphysiological parameter may be acquired based on M second preliminaryphysiological parameters, a third physiological parameter may beacquired based on K third preliminary physiological parameters, and afourth physiological parameter may be acquired based on L fourthpreliminary physiological parameters.

However, the above description is applicable to a method of acquiring aphysiological parameter using a preliminary physiological parameter, andthus a redundant description thereof will be omitted.

The numbers of first to fourth preliminary physiological parameters foracquiring the first to fourth physiological parameters may be the same.

For example, the number of first preliminary physiological parametersfor acquiring the first physiological parameter may be four, the numberof second preliminary physiological parameters for acquiring the secondphysiological parameter may be four, the number of third preliminaryphysiological parameters for acquiring the third physiological parametermay be four, and the number of fourth preliminary physiologicalparameters for acquiring the fourth physiological parameter may be four,but the present invention is not limited thereto.

Also, the numbers of first to fourth preliminary physiologicalparameters for acquiring the first to fourth physiological parametersmay be different from each other.

For example, the number of first preliminary physiological parametersfor acquiring the first physiological parameter may be four, the numberof second preliminary physiological parameters for acquiring the secondphysiological parameter may be two, the number of third preliminaryphysiological parameters for acquiring the third physiological parametermay be two, and the number of fourth preliminary physiologicalparameters for acquiring the fourth physiological parameter may be one,but the present invention is not limited thereto.

Also, the numbers of first to fourth preliminary physiologicalparameters for acquiring the first to fourth physiological parametersmay differ depending on a physiological parameter to be acquired.

For example, when it is assumed that the first physiological parameteris a heart rate, that the second physiological parameter is an oxygensaturation level, that the third physiological parameter is a bloodpressure, and that the fourth physiological parameter is a coretemperature, the number of first preliminary physiological parametersfor acquiring the heart rate may be four, the number of secondpreliminary physiological parameters for acquiring the oxygen saturationlevel may be two, the number of third preliminary physiologicalparameters for acquiring the blood pressure may be two, and the numberof fourth preliminary physiological parameters for acquiring the coretemperature may be one. However, the present invention is not limitedthereto, and the number of preliminary physiological parameters that areto be used as a basis may be set in consideration of a physiologicalparameter to be acquired.

8. Drowsiness Detection Device and Method 8.1 Drowsiness DetectionDevice

A drowsiness detection device described herein may refer to a device fordetecting a subject's drowsiness-related state. In detail, thedrowsiness detection device may detect whether the subject is drowsy,the degree of drowsiness, and the like.

People may feel drowsy during daily life. In some cases, the drowsinessmay pose a safety hazard. For example, when a driver feels drowsy whiledriving, he or she cannot concentrate on driving, which can lead to atraffic accident. In this case, the drowsiness detection device mayprevent a situation which poses a threat to a driver, such as a trafficaccident, by detecting the driver's drowsiness-related state andnotifying the driver of the drowsiness-related state. It will beappreciated that the drowsiness detection device may be used in placesand fields where drowsiness serves as important information, such asreading rooms and infant monitoring, as well as in situations in whichdrowsiness poses a safety hazard.

The drowsiness detection device may detect a drowsiness-related state byusing a contact-type device that involves physical contact with asubject, such as an attachable electrode sensor or a wearable device.Also, the drowsiness detection device may detect a subject'sdrowsiness-related state in a contactless manner by using a camera. Whenthe drowsiness-related state is detected in the contactless manner, thesubject's drowsiness may be detected without the subject beingconstrained by a contact-type device, and thus the drowsiness detectiondevice may have increased user convenience and may also be used invarious places and fields.

The contactless drowsiness detection device may detect adrowsiness-related state by detecting a subject's abnormal state exposedto the outside, for example, by measuring the number of eye blinks of asubject or tracking the position of the pupil of a subject. However,when a subject consciously controls eye blinks or the position of thepupil, it is difficult to accurately detect a drowsiness-related state.When information that appears in a subject's body when the subject feelsdrowsy is used to compensate for the accuracy, it is possible toaccurately detect a drowsiness-related state. Also, when physiologicalinformation that appears in a situation in which a subject does not feeldrowsy but will soon become drowsy is further used, a drowsiness-relatedstate may be detected in advance in an initial stage.

Due to these advantages, a drowsiness detection device 6000 describedherein provided in the specification may detect whether a subject is ina drowsiness-related state and the degree of drowsiness using asubject's physiological parameter acquired in a contactless manner.

According to an embodiment, referring to FIG. 38, the drowsinessdetection device 6000 may include a heartbeat information acquisitionunit 6100 configured to acquire a heart rate from a subject and adrowsiness detection unit 6200 configured to detect the subject'sdrowsiness on the basis of the acquired heart rate of the subject.

It will be appreciated that the drowsiness detection device 6000 of thepresent invention may further include elements other than the heartbeatinformation acquisition unit 6100 and the drowsiness detection unit6200. For example, the drowsiness detection device 6000 may furtherinclude a notification unit 6300 in addition to the heartbeatinformation acquisition unit 6100 and the drowsiness detection unit6200. Here, the notification unit 6300 may provide a notification forwaking up a drowsy subject to the subject or an entity other than thesubject on the basis of a detection result of the drowsiness detectionunit 6200. Here, the entity other than the subject may refer to anadministrator who manages the subject or a third party who is positionedaround the subject.

In an embodiment, the drowsiness detection device 6000 may detect asubject's drowsiness at various time points. That is, some or all of theelements included in the drowsiness detection device 6000 may operate atvarious time points, and each operation or all operations of variousdrowsiness detection methods to be described below may be implemented atvarious time points.

For example, the drowsiness detection device 6000 may detect a subject'sdrowsiness at predetermined intervals (e.g., every minute, every hour,etc.). As another example, the drowsiness detection device 6000 maydetect a subject's drowsiness without following predetermined intervals.As an example, the drowsiness detection device 6000 may detect asubject's drowsiness whenever acquiring information on a subject'sheartbeat, i.e., in real time. Also, the drowsiness detection device6000 may detect a subject's drowsiness in succession or when a signalfor triggering drowsiness detection is input or when a request to detectdrowsiness is received.

In another embodiment, the drowsiness detection device 6000 may startdrowsiness detection when a heart rate is acquired for the first time.For example, the heartbeat information acquisition unit 6100 may acquirea heartbeat for the first time a few seconds or minutes aftermeasurement to acquire a subject's heart rate is started. In this case,the drowsiness detection device 6000 may start to detect the subject'sdrowsiness when a heart rate is acquired for the first time.

Elements included in the drowsiness detection device 6000 according toan embodiment will be described in detail below with reference to FIG.38.

8.1.1 Heartbeat Information Acquisition Unit

In an embodiment, a heartbeat information acquisition unit 6100 mayacquire information on a subject's heart rate, for example, at least oneof a subject's heart rate or a subject's ratio of sympathetic activityto parasympathetic activity.

Here, a heart rate may refer to the number of times a heart beats duringa reference time. Generally, a heart rate may refer to the number ofheartbeats during a reference time of one minute, but the reference timemay be arbitrarily set. For example, the reference time may be set to 30seconds, etc. Also, the ratio of sympathetic activity to parasympatheticactivity is due to a heartbeat signal, where the heartbeat signal mayrefer to a signal that can vary depending on a heartbeat or a signalthat can be estimated to vary depending on a heartbeat. Generally,sympathetic activity and parasympathetic activity are highly associatedwith a subject's drowsiness state. Accordingly, the drowsiness detectiondevice 6000 may detect a subject's drowsiness using the ratio ofsympathetic activity to parasympathetic activity which is due to aheartbeat signal.

Also, the feature of the sympathetic activity may appear in lowfrequency bands of the heartbeat signal, and the feature of theparasympathetic activity may appear in high frequency bands of theheartbeat signal. Herein, accordingly, the ratio of sympathetic activityto parasympathetic activity may be expressed as an LF/HF ratio. LF mayrefer to a characteristic value in a low frequency (LF) band of aheartbeat signal, and HF may refer to a characteristic value in a highfrequency (HF) band of a heartbeat signal.

In an embodiment, a low frequency band and a high frequency band may beclassified based on a reference frequency. As an example, the lowfrequency band and the high frequency band may be classified based on areference frequency of 0.15 Hz. The low frequency band may include asignal with a frequency ranging from 0.04 Hz to 0.15 Hz, and the highfrequency band may include a signal with a frequency ranging from 0.15Hz to 0.4 Hz. It will be appreciated that the reference frequency may beset to frequencies other than 0.15 Hz.

In an embodiment, the heartbeat information acquisition unit 6100 maytransform a heartbeat signal into a signal in a frequency domain byvarious methods such as the Fourier transform. The heartbeat informationacquisition unit 6100 may acquire an LF/HF ratio by extracting alow-frequency band (0.04 to 0.15 Hz) signal and a high-frequency band(0.15 to 0.4 Hz) from the heartbeat signal through the transformation.

In an embodiment, the heartbeat information acquisition unit 6100 mayacquire heart rate information in an invasive or non-invasive manner.

Also, according to another embodiment, the heartbeat informationacquisition unit 6100 may acquire information regarding a heartbeat ratein a contact or contactless manner.

In detail, the heartbeat information acquisition unit 6100 may acquireheart rate information in a contactless manner using a device that doesnot come into physical contact with a subject. For example, theheartbeat information acquisition unit 6100 may acquire an image of asubject using a camera, analyze the acquired image, and acquireinformation on the subject's heart rate. A method and device foracquiring heart rate information in a contactless manner using a camerahave been described above, and thus a detailed description thereof willbe omitted.

Also, it will be appreciated that the heartbeat information acquisitionunit 6100 may acquire heart rate information in a contact manner using adevice that comes into physical contact with a subject. For example, theheartbeat information acquisition unit 6100 may acquire heart rateinformation using an attachable sensor for heartbeat measurement or awearable device including various attachable sensors and opticalsensors. For convenience of description, the elements of the drowsinessdetection device and the drowsiness detection method will be describedbelow, focusing on an example in which heart rate information isacquired in a contactless manner.

In an embodiment, the heartbeat information acquisition unit 6100 mayacquire heart rate information by directly measuring the heart rate. Indetail, the heartbeat information acquisition unit 6100 may include aheartbeat measuring device and may acquire heart rate informationmeasured by the heartbeat measuring device. For example, the heartbeatinformation acquisition unit 6100 may include a camera and may directlymeasure heart rate information by analyzing an image acquired from thecamera.

Also, the heartbeat information acquisition unit 6100 may acquire heartrate information by receiving heart rate information measured by anexternal device. For example, the heart rate information may be measuredby a device including an electrode sensor for heartbeat measurement, andthe heartbeat information acquisition unit 6100 may receive the measuredheart rate information. Also, the heartbeat information acquisition unit6100 may receive the measured heart rate information through a device(e.g., a server) for relaying the heartbeat information acquisition unit6100 and an external device.

In an embodiment, when the drowsiness detection device 6000 is includedin the physiological parameter acquisition device 10, the heartbeatinformation acquisition unit 6100 may read and acquire a heart ratestored in a memory of the physiological parameter acquisition device 10.In another embodiment, when the drowsiness detection device 6000 is notincluded in the physiological parameter acquisition device 10, theheartbeat information acquisition unit 6100 may receive a heart ratemeasured by the physiological parameter acquisition device 10 through acommunication unit (not shown) included in the drowsiness detectiondevice 6000. It will be appreciated that the drowsiness detection device6000 may acquire a heart rate from a device other than the physiologicalparameter acquisition device 10.

8.1.2 Drowsiness Detection Unit

In an embodiment, a drowsiness detection unit 6200 may detect whether asubject is drowsy or the degree of drowsiness felt by a subject on thebasis of heart rate information. In detail, the drowsiness detectionunit 6200 may detect a drowsiness state and a normal state of a subjecton the basis of the heart rate information. Here, the drowsiness statemay be divided into a plurality of levels.

Herein, the drowsiness state may refer to a state in which a subject istemporarily sleeping or a state in which a subject is not sleeping butis likely to sleep within a predetermined period, and the normal state,which is not the drowsiness state, may refer to a state in which asubject is unlikely to fall asleep within a predetermined time.

In an embodiment, the drowsiness state may be divided into a pluralityof levels according to the degree of drowsiness. In some cases, thedrowsiness state may be divided into first to N^(th) levels. Herein, forconvenience of description, the lowest level of the degree of drowsinessis expressed as the first level, and the highest level of the degree ofdrowsiness is expressed as the N^(th) level. For example, the drowsinessstate may be divided into first to third levels. In this case, the firstlevel may refer to the lowest level of the degree of drowsiness, and thethird level may refer to the highest level of the degree of drowsiness.

In an embodiment, a first-level drowsiness state may refer to a state ina subject physically enters a drowsiness state but is not aware ofdrowsiness. For example, in the first-step drowsiness state, heartbeatinformation of a subject objectively indicates that the subject isdrowsy, but the subject himself or herself may not feel drowsy. Herein,this drowsiness state may be expressed as an unconscious drowsinessstate or an unaware drowsiness state.

In an embodiment, a second-level drowsiness state and a third-leveldrowsiness state may refer to states in which a subject physicallyenters a drowsiness state and also is aware of drowsiness. Herein, thesedrowsiness states may be expressed as a conscious drowsiness state or anaware drowsiness state.

Generally, a subject enters the unconscious drowsiness state beforeentering the conscious drowsiness state. When a subject receives anotification about drowsiness after entering the conscious drowsinessstate, there is a high likelihood that a situation dangerous to thesubject will occur because the subject is already drowsy. Also, even ifa notification about the drowsiness state is received before a dangeroussituation occurs, it may take a great deal of time for the subject tocome out of the drowsiness state.

In order to prevent such a dangerous situation, the drowsiness detectiondevice 6000 may detect the unconscious drowsiness state before a subjectenters the conscious drowsiness state and may provide a notificationbased on the detection result to the outside. The drowsiness detectiondevice 6000 may make the subject become aware of the drowsiness statethrough the notification when the subject is in the unconsciousdrowsiness state in which the subject does not feel drowsy.

Accordingly, the detection and notification of the unconsciousdrowsiness state can increase the likelihood that the subject will comeout of the drowsiness state before a dangerous situation occurs and canmore reliably protect the subject from the dangerous situation due tothe drowsiness state.

In an embodiment, the drowsiness detection unit 6200 may measure asubject's drowsiness state using heartbeat information. Herein, adrowsiness state is a state in which a subject is temporarily sleepingor a state in which a subject is likely to sleep within a predeterminedperiod of time immediately before he or she falls asleep. In thedrowsiness state, a parasympathetic nervous system is relatively moreactivated compared to the normal state, and when the parasympatheticnervous system is activated, a heart rate decreases. That is, a changein heart rate may serve as a biomarker capable of estimating thedrowsiness state of the subject. As an example, when a subject entersthe drowsiness state, the heart rate of the subject is decreased below acertain level, and the decreased heart rate may continue for a certaintime. In some cases, the drowsiness detection unit 6200 may estimate thedrowsiness state of the subject through the duration of the decreasedheart rate.

In an embodiment, the drowsiness detection unit 6200 may measure asubject's normal state using heartbeat information. As an example, whena subject enters the normal state, the heart rate of the subject isincreased above a certain level due to the activation of a sympatheticnerve, and the increased heart rate may also continue for a certaintime. In this case, the drowsiness detection unit 6200 may estimate thesubject's normal state through the duration of the increased heart rate.

Also, in another embodiment, the drowsiness detection unit 6200 maymeasure a subject's drowsiness state and normal state using an LF/HFratio. When the subject enters the drowsiness state, the parasympatheticnervous system of the subject may be activated, and the LF value of theheartbeat signal may be decreased relative to the HF value. Also, whenthe subject enters the normal state, the sympathetic nervous system ofthe subject may be activated, and the LF value may be increased relativeto the HF value. Accordingly, the drowsiness detection unit 6200 mayestimate the subject's drowsiness state and normal state through anLF/HF ratio.

In still another embodiment, the drowsiness detection unit 6200 mayestimate the subject's drowsiness state or normal state in considerationof both of the change in heart rate and the LF/HF ratio.

The drowsiness detection method will be described in detail below.

8.1.3 Notification Unit

In an embodiment, a notification unit 6300 may provide detecteddrowsiness-state-related information to the outside. In detail, thenotification unit 6300 may provide information on whether a subject isdrowsy, a drowsiness level, and also various pieces of detecteddrowsiness-state-related information. For example, the notification unit6300 may provide a duration for which the drowsiness state ismaintained, a time at which each level of the drowsiness state isentered, a time at which the drowsiness state is exited, and the like tothe outside.

In an embodiment, the notification unit 6300 may providedrowsiness-state-related information to a subject. For example, thenotification unit 6300 may notify a subject at an intensitycorresponding to the level of the drowsiness state. The notification mayinduce recovery from the drowsiness state by applying a certain stimulusto a drowsy subject.

In another embodiment, the notification unit 6300 may provide thedrowsiness-state-related information to an entity other than thesubject. For example, the notification unit 6300 may providedrowsiness-state-related information to an administrator or a server.Here, an administrator may refer to an administrator of a subject or anentity who manages a drowsiness state of a subject. The provision ofsuch information can help the administrator to effectively manage thesafety of the subject on the basis of the drowsiness-state-relatedinformation.

In an embodiment, the notification unit 6300 may provide information ondrowsiness states detected at all the levels to the outside. Forexample, the notification unit 6300 may provide drowsiness-state-relatedinformation regarding first-level drowsiness state, second-leveldrowsiness state, and third-level drowsiness state to a subject.

Also, in another embodiment, the notification unit 6300 may provideinformation on only at least some of the drowsiness states detected atall the levels to the outside.

For example, the notification unit 6300 may provide an administratorwith information regarding a time and a level at which the third-leveldrowsiness state is detected only when the third-level drowsiness stateis detected. When notifications about the drowsiness states at all thelevels interfere with the subject, the notification unit 6300 canincrease the work efficiency of the subject by providing onlyinformation on a high-risk drowsiness level (e.g., information on athird-level drowsiness state) and not providing information on alow-risk drowsiness level (e.g., information on a first-level drowsinessstate).

Also, in an embodiment, when a subject is detected as being in a normalstate, the notification unit 6300 may provide information on thedetected normal state. In detail, the notification unit 6300 may providevarious pieces of information associated with the detected normal state,including whether the subject is in the normal state. For example, thenotification unit 6300 may provide a duration for which the normal stateis maintained, a time at which the normal state is entered, etc.

In an embodiment, the notification unit 6300 may providenormal-state-related information to the subject. For example, when adrowsy subject has recovered to the normal state, the notification unit6300 may output a message for notifying the subject that he or she hasrecovered to the normal state.

In another embodiment, the notification unit 6300 may provide thenormal-state-related information to an entity other than the subject.For example, the notification unit 6300 may provide thenormal-state-related information to an administrator or a server. Theprovision of such information can help the administrator to effectivelyrecognize that the subject is in a safe situation.

In an embodiment, the notification unit 6300 may generate a notificationsignal indicating the drowsiness-state-related information and providethe notification signal to an output unit (not shown) so that thenormal-state-related information and the drowsiness-state-relatedinformation are output to the outside of the drowsiness detectiondevice.

Here, the output unit is a device configured to provide information tothe outside. For example, the output unit may include at least one of adisplay unit configured to display information to the outside in avisual manner, an audio unit configured to provide information in anauditory manner, and a tactile stimulation unit configured to provideinformation in a tactile manner. The output unit may be included in thedrowsiness detection device 6000 or an external device of the drowsinessdetection device 6000.

When the output unit is included in a device outside the drowsinessdetection device 6000, the drowsiness detection device 6000 may furtherinclude a communication unit (not shown), and the notification unit 6300may provide a notification signal to the output unit through thecommunication unit. The output unit may receive the notification signalprovided by the notification unit 6300 and output drowsiness-relatedinformation to the outside according to the notification signal.

For example, the notification unit 6300 may provide an auditorynotification through the audio unit. Such an auditory notification iseffective because it can give an immediate stimulus to a notificationrecipient and change the intensity of the stimulus according to soundintensity.

As another example, the notification unit 6300 may provide anotification by displaying a message on a display through the displayunit. Such a visual notification can minimize a situation in which thenotification interferes with an entity other than the notificationrecipient.

As another example, the notification unit 6300 may provide anotification through the tactile stimulation unit, for example,vibration, electric stimulus, a thermal element, or a cooling element.Such a tactile notification is effective because it may not interferewith an entity other than the notification recipient, can provide animmediate stimulus, and change the intensity of the stimulus accordingto stimulus intensity.

Also, in an embodiment, the notification unit 6300 may provide a subjectwith notifications of different types and/or intensities depending onthe detected level of the drowsiness state. In this case, thenotification unit 6300 may provide a notification of a type suitable forrecognizing that the subject is in the drowsiness state and/or of anintensity necessary for the subject to come out of the drowsiness state.

For example, when the first-level drowsiness state, which is the lowdegree of drowsiness state, is detected, the notification unit 6300 maymake the subject recognize that he or she is drowsy by notifying him orher using a visual message. Also, when the third-level drowsiness state,which is the high degree of drowsiness state, is detected, thenotification unit 6300 may make the subject come out of the drowsinessstate by notifying; him or her using electrical stimulation.

As another example, the notification unit 6300 may provide a lowintensity alarm to the subject when the first-level drowsiness state,which is the low degree of drowsiness state, is detected, and thenotification unit 6300 may provide a high intensity alarm to the subjectwhen the third-level drowsiness state is detected.

It will be appreciated that, in some embodiments, the notification unit6300 may provide a notification of the same type and/or intensityregardless of the level of the drowsiness state detected by the subject.

Also, the notification unit 6300 may notify at least one of a subjectand an entity other than the subject according to the detected level ofthe drowsiness state.

In an embodiment, the notification unit 6300 may notify the subject whenthe first-level drowsiness state and the second-level drowsiness stateare detected. In this case, the notification unit 6300 may provide thesubject with at least one of an auditory notification, a visualnotification, and a tactile notification.

Also, when the third-level drowsiness state is detected, thenotification unit 6300 may notify an entity other than a subject as wellas the subject. In detail, the notification unit 6300 may notify theentity other than the subject through a device or server for relayingthe notification unit 6300 and the entity other than the subject. Forexample, when the entity other than the subject is an administrator whomanages the subject, such a notification may be effective in a situationin which the subject turns off or ignores a notification providedthrough an output unit. A drowsiness detection method performed by thedrowsiness detection device 6000 will be described in detail below.

8.2 Heart Rate-Based Drowsiness Detection Method

A heart rate-based drowsiness detection method according to anembodiment will be described below with reference to FIGS. 39 to 43.

In an embodiment, the heart rate-based drowsiness detection method mayinclude a heart rate acquisition operation (S6110), an operation ofcomparing an acquired heart rate of a subject to a reference heart rate(S6120), a duration measurement operation for measuring a duration forwhich a heart rate is changed according to a result of the comparison(S6130), and a level-specific drowsiness detection operation fordetecting drowsiness for each level on the basis of the measuredduration of the change in heart rate (S6140).

In the heart rate acquisition operation (S6110), a drowsiness detectiondevice 6000 may acquire a subjects heart rate using a heartbeatinformation acquisition unit 6100.

In an embodiment, the drowsiness detection device 6000 may acquire asubject's heart rate at predetermined intervals. Here, the predeterminedinterval may include various intervals such as one second, one minute,and five minutes. Also, it will be appreciated that the predeterminedinterval may be fixed or variable.

In another embodiment, the drowsiness detection device 6000 may acquirea subject's heart rate without following predetermined intervals. Forexample, the drowsiness detection device 6000 may detect a subject'sheart rate whenever the physiological parameter acquisition device 10 orthe like measures the subject's heart rate, i.e., in real time.

It will be appreciated that when a subject's heart rate is measured bythe physiological parameter acquisition device 10 or the like atpredetermined intervals, the drowsiness detection device 6000 mayacquire the subject's heart rate on the basis of the predeterminedintervals at which the physiological parameter acquisition device 10 orthe like measure the heart rate.

As another example, the drowsiness detection device 6000 may request thephysiological parameter acquisition device 10 or the like to provide aheart rate upon an external input or request and may acquire a subject'sheart rate each time the request is made.

In an embodiment, the drowsiness detection device 6000 may acquire asubject's average heart rate corresponding to a certain time period. Thetime period may be arbitrarily determined. Here, the drowsinessdetection device 6000 may calculate an average heart rate correspondingto a certain time period using the subject's heart rate received from anexternal device. Also, the drowsiness detection device 6000 may receivethe subject's average heart rate corresponding to a certain time periodfrom an external device.

An effect that can be obtained when an average heart rate is acquired asa subject's heart rate will be described below with reference to FIG.40.

In an embodiment, referring to FIG: 40, the drowsiness detection device6000 may acquire heart rate from which noise is removed by acquiring anaverage heart rate 6003.

Here, noise represents errors caused by various causes such as asubject's motion or external light. When noise is contained in a heartrate, it may be difficult for the corresponding heart rate to accuratelyreflect the subject's state.

For example, it is assumed that the heartbeat information acquisitionunit 6100 acquires a heart rate every second from one second to tenseconds. When noise occurs after a lapse of five seconds, the drowsinessdetection device 6000 may not accurately detect drowsiness, However,when the heartbeat information acquisition unit 6100 acquires theaverage heart rate 6003, the drowsiness detection device 6000 may moreaccurately detect drowsiness because deviations between noise and othervalues can be corrected while the average heart rate 6003 is calculated.

A method of the heartbeat information acquisition unit 6100 acquiring asubject's heart rate has been described in Section 8.1.1, and thus adetailed description thereof will be omitted.

The operation of comparing a subject's heart rate to a reference heartrate (S6120) and the duration measurement operation for measuring aduration for which the acquired heart rate of the subject is decreasedor increased and then maintained will be described below with referenceto FIG. 41.

According to an embodiment, referring to FIG. 41, the drowsinessdetection device 6000 may compare a subject's heart rate 6001 acquiredby the heartbeat information acquisition unit 6100 to a reference heartrate 6002 in the operation of comparing the subject's heart rate to thereference heart rate (S6120).

Here, the reference heart rate 6002 may refer to a heart rate that isset as a reference for distinguishing between a drowsiness state and anormal state. As described above, due to the activation of aparasympathetic nervous system, a heart rate in the drowsiness statedecreases below a heart rate in the normal state. Accordingly, in orderto distinguish between the drowsiness state and the normal state, aheart rate which is the same or similar to the decreased heart rate maybe set as the reference heart rate 6002. In this case, when the heartrate of the subject is equal to or below the reference heart rate 6002,the drowsiness detection device 6000 may determine that the subject isin the drowsiness state. When the heart rate 6001 of the subject isequal to or above the reference heart rate 6002, the drowsinessdetection device 6000 may determine that the subject is in the normalstate.

Various methods for setting the reference heart rate 6002 will bedescribed below.

In an embodiment, the drowsiness detection device 6000 may set thereference heart rate 6002 using the mean of heart rates acquired duringa certain time after the heart rate is acquired for the first time. Thecertain time may be set to various values such as one minute or fiveminutes. Generally, there is a high possibility that the subject is notin the drowsiness state but in the normal state during a certain timeafter the subject's heart rate is acquired for the first time.Accordingly, the drowsiness detection device 6000 may acquire theaverage heart rate of the subject acquired during the certain time andmay acquire a value obtained by multiplying the average heart rate by apredetermined value a (e.g., a indicates a real number less than orequal to 1 and, as a specific example, is 0.9) as the reference heartrate 6002. The predetermined value a is not limited to the aboveexample.

In another embodiment, the drowsiness detection device 6000 may set thereference heart rate 6002 using a resting heart rate and/or an activeheart rate. In detail, the resting heart rate may refer to a heart ratewhen the subject is not moving (or when the amount of movement issmall). Herein, a heart rate when a subject is awake but not moving (orthe amount of movement is small) is defined as a first resting heartrate, and a heart rate when a subject is sleeping is defined as a secondresting heart rate.

Also, the active heart rate may refer to a heart rate when the subjectis actively moving.

In an embodiment, the drowsiness detection device 6000 may acquire aheart rate measured in a situation in which the subject is awake but notmoving (or the amount of movement is small) as a first resting heartrate. For example, when a subject is in an environment in which he orshe cannot be actively moving (e.g., the subject seats on the driverseat), the drowsiness detection device 6000 may assume that the subjectis awake but not moving during a certain time after a heart rate isacquired for the first time, and then may set a heart rate measuredduring the certain time as a first resting heart rate.

Also, the drowsiness detection device 6000 may acquire, as a secondresting heart rate, a heart rate measured in a situation in which thesubject is recognized as being sleeping and may acquire, as an activeheart rate, a heart rate measured in a situation in which the subject isrecognized as being actively moving.

Also, the drowsiness detection device 6000 may acquire the first restingheart rate, the second resting heart rate, and the active heart rateusing a camera or a wearable device.

For example, the drowsiness detection device 600 may use a camera toacquire information on whether a subject is moving and whether the pupilof a subject is being tracked. As an example, the drowsiness detectiondevice 6000 may acquire, as the active heart rate, a heart rate measuredwhen a subject is moving, Also, by using a camera, the drowsinessdetection device 6000 may acquire, as the first resting heart rate, aheart rate measured when the pupil of a subject is continuously trackedeven if the subject does not move and may acquire, as the second restingheart rate, a heart rate measured when the pupil of a subject is notcontinuously tracked while the subject does not move.

As another example, the drowsiness detection device 600 may use awearable device to acquire information on whether a subject is movingand whether a body part (e.g., a wrist) on which the wearable device isworn is being moved. As an example, the drowsiness detection device 6000may use a wearable device to acquire, as the active heart rate, a heartrate measured when the movement of a subject is detected. Also, thedrowsiness detection device 6000 may acquire, as the first resting heartrate, a heart rate measured when the movement of the body part on whichthe wearable device is worn is detected even if the movement of thesubject is not detected and may acquire, as the second resting heartrate, a heart rate measured when the movement of the body part is notdetected during a certain time while the movement of the subject is notdetected.

Also, the drowsiness detection device 6000 may preset the first restingheart rate, the second resting heart rate, and the active heart ratebefore starting drowsiness detection or may receive the first restingheart rate, the second resting heart rate, and the active heart ratefrom the outside before or after starting drowsiness detection.

According to an embodiment, the reference heart rate 6002 may be setbased on the first resting heart rate. In detail, the reference heartrate 6002 may be set to a certain proportion of the first resting heartrate. For example, the reference heart rate 6002 may be calculated bymultiplying the first resting heart rate, by a predetermined value a(e.g., a indicates a real number less than or equal to 1 and, as aspecific example, is 0.9). It will be appreciated that the predeterminedvalue a is not limited to the above example.

According to another embodiment, the reference heart rate 6002 may beset based on the second resting heart rate. In detail, the referenceheart rate 6002 may be set to be the same as the second resting heartrate or may be calculated by multiplying the second resting heart rateby a predetermined value b (e.g., b indicates a real number greater thanone, and as a specific example, is 1.1). It will be appreciated that thepredetermined value b is not limited to the above example.

Also, according to another embodiment, the reference heart rate 6002 maybe set based on the active heart rate. In detail, the reference heartrate 6002 may be calculated by multiplying the active heart rate by apredetermined value c (e.g., c is a real number less than or equal to 1and, as a specific example, is 0.8). Also, the predetermined value c maybe set to be lower than the predetermined value a, considering thatgenerally, the active heart rate is higher than the first resting heartrate. It will be appreciated that the predetermined value c is notlimited to the above example.

It will also be appreciated that a method of setting the reference heartrate 6002 is not limited to a method of calculating the reference heartrate 6002 as a certain proportion of the first resting heart rate, thesecond resting heart rate, or the active heart rate and that variousmathematical operations such as addition and subtraction are applicable.

In detail, in an embodiment, the drowsiness detection device 6000 mayset the reference heart rate 6002 by adding or subtracting apredetermined value to or from the first resting heart rate, the secondresting heart rate, or the active heart rate. For example, when thefirst resting heart rate is 75, the drowsiness detection device 6000 mayset the reference heart rate 6002 to 65 (=first resting heart rate−10)(beats per minute) on the basis of the first resting heart rate. Forexample, when the second resting heart rate is 65, the drowsinessdetection device 6000 may set the reference heart rate 6002 to 75(=second resting heart rate+10) (heats per minute) on the basis of thesecond resting heart rate.

Also, in another embodiment, the drowsiness detection device 600 maychange the previous reference heart rate 6002 to a new reference heartrate 6002. That is, the drowsiness detection device 6000 may update theprevious reference heart rate 6002 to set the new reference heart rate6002.

For example, the drowsiness detection device 6000 may set the referenceheart rate 6002 on the basis of heart rates acquired during a certaintime after the heart rate of the subject is acquired for the first timeas described above. The drowsiness detection device 6000 may detect thenormal state on the basis of the subject's heart rates acquired afterthe certain time and may change an average heart rate in a time intervalin which the normal state is detected to a new reference heart rate6002. Also, the drowsiness detection device 6000 may change the averageheart rate to the new reference heart rate 6002 on the basis of thefirst resting heart rate, the second resting heart rate, and the activeheart rate acquired for the same subject after the certain time. Thus,the drowsiness detection device 6000 may set a reference heart rate 6002reflecting the latest state of the subject and thus may detect thedrowsiness of the subject on the basis of the more accurate referenceheart rate 6002.

As another example, the drowsiness detection device 6000 may preset thereference heart rate 6002 on the basis of the first resting heart rate,the second resting heart rate, and the active heart rate as describedabove. In this case, after the reference heart rate 6002 is set based onthe first resting heart rate, the second resting heart rate, or theactive heart rate, the drowsiness detection device 6000 may newlyacquire a first resting heart rate, a second resting heart rate, or anactive heart rate during a predetermined time. The drowsiness detectiondevice 6000 may set a new reference heart rate 6002 using the newlyacquired first resting heart rate, second resting heart rate, or activeheart rate according to the above-described method of setting thereference heart rate 6002.

According to an embodiment, referring to FIG. 41, in the operation ofcomparing the subject's heart rate 6001 to the reference heart rate 6002(S6120), the drowsiness detection device 6000 may compare the subject'sheart rate 6001 to the reference heart rate 6002.

In an embodiment, the drowsiness detection device 6000 may compare thesubject's heart rate 6001 to the reference heart rate 6002 to detect atime point t1 at which the subject's heart rate 6001 is less than (orgreater than) or equal to the reference heart rate 6002. The detectedtime point t1 may be a starting point for measuring a drowsiness stateand normal state for each level.

That is, in the above embodiment, the detection of the time point t1 atwhich the subject's heart rate 6001 is less than (or greater than) orequal to the reference heart rate 6002 may act as a trigger formeasuring the drowsiness state and the normal state for each level.

In the duration measurement operation S6130, the drowsiness detectiondevice 6000 may measure a duration for which the subject's heart rate6001 is less than or equal to the reference heart rate 6002 with respectto a time point when the subject's heart rate 6001 becomes less than orequal to the reference heart rate 6002. Also, the drowsiness detectiondevice 6000 may measure a duration for which the subject's heart rate6001 is greater than or equal to the reference heart rate 6002 withrespect to a time point when the subject's heart rate 6001 is greaterthan or equal to the reference heart rate 6002.

A heart rate may vary due to several physical factors in addition todrowsiness. Among various physical factors, the decrease or increase inheart rate due to drowsiness is maintained. Thus, the decreased orincreased heart rate being maintained for several seconds to minutes maybe a good biomarker indicating a drowsiness state.

For example, when a person's tension is relieved, a heart rate istemporarily decreased, but the decreased heart rate can be easilyrecovered to normal. When the drowsiness detection device 600 detects adrowsiness state on the basis of a duration for which the heart rate isdecreased and maintained, the drowsiness detection device 6000 may avoiddetecting, as the drowsiness state, a situation in which the heart rateis temporarily decreased due to the tension relief. Thus, the drowsinessdetection device 6000 may more accurately detect the drowsiness state.

Also, as the subject enters a higher degree of drowsiness state, theduration of the heart rate being decreased due to drowsiness may beincreased. Accordingly, by measuring a duration for which the decreasedheart rate is maintained, the drowsiness detection device 6000 maydetermine which level of the drowsiness state the subject has entered.That is, by classifying the level of the drowsiness state according tothe length of the duration, the drowsiness detection device 6000 maymore specifically detect the subject's drowsiness state.

According to an embodiment, referring to FIG. 41, the drowsinessdetection device 6000 may measure a duration for which the subject'sheart rate 6001 is less than or equal to the reference heart rate 6002with respect to a time point t1 at which the subject's heart rate 6001becomes less than or equal to the reference heart, rate 6002.

For example, the reference heart rate 6002 may be set to 72 (beats perminute) corresponding to 90% of the first resting heart rate. Here, whenthe heart rate of the subject is 74 (beats per minute) at a first timepoint and is 72 (beats per minute) at a second time point immediatelyafter the first time point, the drowsiness detection device 6000 maymeasure a duration for which the heart rate of the subject at the secondtime point is less than or equal to 72 (beats per minute).

It will be appreciated that, according to another embodiment, thedrowsiness detection device 6000 may measure a duration for which thesubject's heart rate is greater than or equal to the reference heartrate with respect to a time point at which the subject's heart ratebecomes greater than or equal to the reference heart rate.

For example, the reference heart rate 6002 may be set to 72 (beats perminute) corresponding to 90% of the first resting heart rate. Here, whenthe heart rate 6001 of the subject is 70 (beats per minute) at a thirdtime point and is 72 (beats per minute) at a fourth time pointimmediately after the third time point, the drowsiness detection device6000 may measure a duration for which the heart rate 6001 of the subjectis greater than or equal to 72 (beats per minute) after the fourth timepoint.

According to an embodiment, referring to FIG. 42, when measuring aduration for which the subject's heart rate 6001 is less than (orgreater than) or equal to the reference heart rate 6002, the drowsinessdetection device 6000 may correct a time interval Δtn in which a heartrate containing noise is measured and then may measure the duration. Asan example, the drowsiness detection device 6000 may correct a heartrate detected as noise on the basis of previously measured heart rates.As another example, the time interval Δtn in which the heart ratecontaining noise is measured may be excluded when the duration ismeasured.

Here, noise represents errors caused by various causes such as asubject's motion or external light, and it may be difficult for a heartrate containing noise to reflect the subject's state. Generally, a heartrate containing noise may be temporarily acquired and greatly differentfrom a previously acquired heart rate of the subject and a subsequentlyacquired heart rate of the subject.

Based on this fact, according to an embodiment, when an acquired heartrate has a deviation greater than or equal to a certain value from thepreviously acquired heart rate 6001 of the subject and/or thesubsequently acquired heart rate 6001 of the subject, the drowsinessdetection device 6000 may detect the subjects heart rate 6001 as a heartrate containing noise. For example, when acquiring a heart rate having adeviation of 20 or more from a previously detected heart rate and/or asubsequently acquired heart rate during a time interval of 3 seconds orless, the drowsiness detection device 600 may detect the subject's heartrate 6001 acquired for three seconds as a heart rate containing noise.Here, a time at which the heart rate containing noise is acquired and adeviation between the heart rate containing noise and other heart ratesare not limited to specific values.

According to an embodiment, although the heart rate containing noise isa heart rate greater than or equal to the reference heart rate 6002,when measuring a duration for which the subject's heart rate 6001 isless than or equal to the reference heart rate 6002, the drowsinessdetection device 6000 may regard the noise as a heart rate less than orequal to the reference heart rate and then measure the duration.

According to an embodiment, although the heart rate containing noise isa heart rate less than or equal to the reference heart rate, whenmeasuring a duration for which the subject's heart rate is greater thanor equal to the reference heart rate, the drowsiness detection device6000 may regard the noise as a heart rate less than or equal to thereference heart rate and then measure the duration.

According to an embodiment, although the heart rate containing noise isa heart rate less than or equal to the reference heart rate, whenmeasuring a duration for which the subject's heart rate is less than orequal to the reference heart rate, the drowsiness detection device 6000may exclude the time interval Δtn in which the heart rate containingnoise is measured from the measurement.

According to an embodiment, although the heart rate containing noise isa heart rate less than or equal to the reference heart rate, whenmeasuring a duration for which the subject's heart rate is greater thanor equal to the reference heart rate, the drowsiness detection device6000 may exclude the time interval Δtn in which the heart ratecontaining noise is measured from the measurement.

According to an embodiment, referring to FIG. 41, in the level-specificdrowsiness state detection operation S6140, the drowsiness detectiondevice 6000 may detect a drowsiness state or a normal state through acomparison between a measured duration and a reference duration.

The drowsiness detection device 6000 may measure a duration for whichthe subject's heart rate 6001 is less than or equal to the referenceheart rate and may detect a drowsiness state when the measured durationreaches a reference duration Δta, Δtb, or Δtc. In this case, thereference duration Δta, Δtb, or Δtc may include a plurality of referencedurations. For example, the number of reference durations Δta, Δtb, andΔtc may be three. In this case, a first reference duration Δta may beset to 30 seconds, a second reference duration Δtb may be set to 60seconds, and a third reference duration Δtc may be set to 90 seconds.

In this case, when the duration for which the subject's heart rate 6001is less than or equal to the reference heart rate 6002 reaches 30seconds, the drowsiness detection device 6000 may detect that thesubject is in the first-level drowsiness state.

In this case, when the duration for which the subject's heart rate 6001is less than or equal to the reference heart rate 6002 reaches 60seconds, the drowsiness detection device 6000 may detect that thesubject is in the second-level drowsiness state.

In this case, when the duration for which the subject's heart rate 6001is less than or equal to the reference heart rate 6002 reaches 90seconds, the drowsiness detection device 6000 may detect that thesubject is in the third-level drowsiness state.

Also, before detecting a different-level drowsiness state or a normalstate for a subject detected as being in a drowsiness state, thedrowsiness detection device 6000 may maintain the detected drowsinessstate.

In another embodiment, when the duration for which the subject's heartrate 6001 is less than or equal to the reference heart rate 6002 doesnot reach the first reference duration Δta, the drowsiness detectiondevice 6000 may detect that the subject is in the normal state.

For example, when it is assumed that the first reference duration Δta is30 seconds and that the subject's heart rate 6001 has recovered to aheart rate greater than or equal to the reference heart rate after theheart rate 6001 is maintained less than or equal to the reference heartrate for 10 seconds, the drowsiness detection device 6000 may detectthat the subject is in the normal state.

A method of detecting recovery from a drowsiness state on the basis of aheart rate will be described below with reference to FIG. 43.

Also, in the drowsiness-related state detection operation (S6140), thedrowsiness detection device 6000 may detect that the subject hasrecovered from the drowsiness state when the measured duration reaches arecovery reference duration Δtd, Δte, or Δtf.

The recovery from the drowsiness state may refer to a state in which asubject comes out of a detected drowsiness state after entering thedrowsiness state. In detail, the recovery from the drowsiness state mayrefer to a situation in which a drowsiness state at a level lower than apredetermined level is detected after a drowsiness state at thepredetermined level is detected, Also, the recovery from the drowsinessstate may refer to a situation in which the normal state is detectedafter the drowsiness state is detected.

According to an embodiment, when a duration, for which the heart rate6001 of a drowsy subject is greater than or equal to a recoveryreference heart rate 6004, reaches a recovery reference duration Δtd,Δte, or Δtf, the drowsiness detection device 6000 may detect therecovery of the subject from the drowsiness state.

In detail, according to an embodiment, referring to FIG. 46A, thedrowsiness detection device 6000 may measure a duration for which thesubject's heart rate 6001 is greater than or equal to the recoveryreference heart rate 6004 from the time point t1 and may detect that thesubject has recovered from the drowsiness state at time points t1, t2,and t3 at which the duration reaches the recovery reference duration.

Here, the recovery reference duration may be set to various values. Forexample, the recovery reference duration may be set to various unitssuch as several seconds, tens of seconds, several minutes, and tens ofminutes.

For example, the recovery reference duration may be set to 30 seconds.In this case, when a duration, for which a heart rate of a subjectdetected as being in a third-level drowsiness state is greater than orequal to the recovery reference heart rate 6004, reaches 30, thedrowsiness detection device 6000 may detect that the subject hasrecovered to the normal state.

Here, there may be a plurality of recovery reference durations. Forexample, there are three recovery reference durations. A first recoveryreference duration may be set to 20 seconds, a second recovery referenceduration may be set to 40 seconds, and a third recovery referenceduration may be set to 60 seconds. Here, the first to third recoveryreference durations are limited to the values proposed in the aboveexample.

According to an embodiment, referring to FIG. 43A, the drowsinessdetection device 6000 may measure a duration, for which a subject'sheart rate 6001 detected as being in the third-level drowsiness state isgreater than or equal to the recovery reference heart rate 6004, from atime point t1 at which the subject's heart rate 6001 becomes greaterthan or equal to the recovery reference heart rate 6004 and may detectthat the subject has recovered to the second-level drowsiness state at atime point t2 at which the duration reaches the first recovery referenceduration.

Also, after the subject has recovered from the third-level drowsinessstate to the second-level drowsiness state, the drowsiness detectiondevice 6000 may measure a duration for which a heart rate 6001 of asubject is greater than or equal to the recovery reference heart rate6004 from a time point t1 at which the subject's heart rate 6001 becomesgreater than or equal to the recovery reference heart rate 6004 and maydetect that the subject has recovered to the first-level drowsinessstate at a time point t3 at which the duration reaches the secondrecovery reference duration.

However, according to an embodiment, referring to FIG. 43B, after thesubject has recovered from the third-level drowsiness state to thesecond-level drowsiness state and before a duration, for which thesubject's heart rate 6001 is greater than or equal to the recoveryreference heart rate 6004 from a time point t4 at which the subject'sheart rate 6001 is greater than or equal to the recovery reference heartrate 6004, reaches the second recovery duration, the drowsinessdetection device 6000 may detect a time point t6 at which the heart rate6001 of the subject is less than or equal to the recovery referenceheart rate 6004. In this case, the drowsiness detection device 6000 maymaintain the drowsiness state of the subject in the second-leveldrowsiness state and also reset the drowsiness state of the subject tothe third-level drowsiness state.

It will be appreciated that the drowsiness detection device 6000 maydetect recovery from the drowsiness state on the basis of a duration forwhich an average heart rate 6003 of the subject is greater than or equalto the recovery reference heart rate 6004.

The recovery reference heart rate 6004 may be the same as or differentfrom the reference heart rate 6002 for detecting a drowsiness state.

The recovery reference duration may be the same as or different from aduration for detecting a drowsiness state.

8.3 LF/HT Ratio-Based Drowsiness Detection Method

An LF/HF ratio-based drowsiness detection method according to anembodiment will be described below with reference to FIGS. 44 to 47.

In an embodiment, the LF/FH-based drowsiness detection method mayinclude an operation of acquiring a subject's LF/HF ratio 6005 (S6210),an operation of comparing the subject's LF/HF ratio 6005 to a referenceLF/HF ratio (S6220), and an operation of detecting drowsiness for eachlevel on the basis of a result of the comparison (S6230).

In some embodiments, the drowsiness detection method may further includeproviding a notification corresponding to a drowsy level to the outside.

In the operation of acquiring an LF/HF ratio (S6210), the heartbeatinformation acquisition unit 6100 may acquire the subject's LF/HF ratio6005.

In an embodiment, the drowsiness detection device 6000 may acquire thesubject's LF/HF ratio 6005 at predetermined intervals. In anotherembodiment, the drowsiness detection device 6000 may acquire thesubject's LF/HF ratio 6005 without following predetermined intervals.

It will be appreciated that when a subject's LF/HF ratio 6005 ismeasured by the physiological parameter acquisition device 10 or thelike at predetermined intervals, the drowsiness detection device 6000may acquire the subject's LF/HF ratio 6005 on the basis of thepredetermined intervals at which the physiological parameter acquisitiondevice 10 or the like measure the LF/HF ratio.

As another example, the drowsiness detection device 6000 may request thephysiological parameter acquisition device 10 or the like to provide anLF/HF ratio upon an external input or request and may acquire asubject's LF/HF ratio each time the request is made.

In an embodiment, the drowsiness detection device 6000 may acquire asubject's average LF/HF ratio for a certain time period.

The drowsiness detection device 6000 can correct noise by acquiring theaverage LF/HF ratio, and thus can have an effect of accurately detectingdrowsiness. This effect is the same as an effect obtainable when anaverage heart rate is acquired and thus a detailed description thereofwill be omitted.

According to an embodiment, referring to FIG. 45, in the operation ofcomparing an acquired LF/HF ratio 6005 of a subject to reference LF/HFratios 6006, 6007, and 6008 (S6220), the drowsiness detection device6000 may compare the acquired LF/HF ratio 6005 of the subject to areference LF/HF ratio.

Here, the reference LF/HF ratios 6006, 6007, and 6008 may refer tovalues that are set as a reference for distinguishing between adrowsiness state and a normal state. As described above, an LF/HF ratioin the drowsiness state may be smaller than an LF/HF ratio in the normalstate according to the activation of a parasympathetic nervous system.

In an embodiment, the drowsiness detection device 6000 may set thereference LF/HF ratios 6006, 6007, and 6008 using the mean of LF/HFratios of a heartbeat signal acquired during a certain time from a timepoint at which the LF/HF ratio of the heartbeat signal is acquired forthe first time. The certain time may be set to various values such asone minute or five minutes. Generally, there is a high possibility thatthe subject is in the normal state during a certain time after thesubject's LF/HF ratio 6005 is acquired for the first time. Accordingly,the drowsiness detection device 6000 may acquire an average LF/HF ratioof the subject acquired during the certain time and may acquire a valueobtained by multiplying the average LF/HF ratio by a predetermined valuea (e.g., a indicates a real number less than or equal to one and, as aspecific example, is 0.9) as a reference LF/HF ratio. Also, thedrowsiness detection device 6000 may set a plurality of reference LF/HFratios using different predetermined real numbers. The predeterminedvalue a is not limited to the above example.

In another embodiment, the drowsiness detection device 6000 may set areference LF/HF ratio 6006, 6007, or 6008 using a resting LF/HF ratioand/or an active LF/HF ratio. In this case, there may be a plurality ofreference LF/HF ratios 6006, 6007, and 6008.

In detail, the drowsiness detection device 6000 may use a camera or awearable device to obtain a first resting LF/HF ratio based on a firstresting heart rate, a second resting LF/HF ratio based on a secondresting heart rate, and an active LF/HF ratio based on an active heartrate.

According to an embodiment, the reference LF/HF ratios 6006, 6007, and6008 may be set based on the first resting LF/HF ratio. In detail, thereference LF/HF ratios 6006, 6007, and 6008 may be set to certainproportions of the first resting LF/HF ratio. For example, the referenceLF/HF ratios 6006, 6007, and 6008 may include a first reference LF/HFratio 6006 obtained by multiplying the first resting LF/HF ratio by 0.9,which is a predetermined value, a second reference LF/HF ratio 6007obtained by multiplying the first resting LF/HF ratio by 0.8, and athird reference LF/HF ratio 6008 obtained by multiplying the firstresting LF/HF ratio by 0.7. It will be appreciated that thepredetermined value is not limited to the above example.

According to another embodiment, the reference LF/HF ratios 6006, 6007,and 6008 may be set based on the second resting LF/HF ratio. In detail,the reference LF/HF ratios 6006, 6007, and 6008 may be set to certainproportions of the second resting LF/HF ratio. For example, thereference LF/HF ratios 6006, 6007, and 6008 may include a firstreference LF/HF ratio 6006 obtained by multiplying the second restingLF/HF ratio by 1.5, which is a predetermined value, a second referenceLF/HF ratio 6007 obtained by multiplying the second resting LF/HF ratioby 1.3, and a third reference LF/HF ratio 6008 obtained by multiplyingthe second resting LF/HF ratio by 1.1. It will be appreciated that thepredetermined value is not limited to the above example.

Also, according to another embodiment, the reference LF/HF ratios 6006,6007, and 6008 may be set based on the active LF/HF ratio. In detail,the reference LF/HF ratios 6006, 6007, and 6008 may include a firstreference LF/HF ratio 6006 obtained by multiplying the active LF/HFratio by 0.8, which is a predetermined value, a second reference LF/HFratio 6007 obtained by multiplying the active LF/HF ratio by 0.7, and athird reference LF/HF ratio 6008 obtained by multiplying the activeLF/HF ratio by 0.6. It will be appreciated that the predetermined valueis not limited to the above example.

It will also be appreciated that a method of setting the reference LF/HFratios 6006, 6007, and 6008 is not limited to a method of calculatingthe reference LF/HF ratios 6006, 6007, and 6008 as certain proportionsof the first resting LF/HF ratio, the second resting LF/HF ratio, andthe active LF/HF ratio and various mathematical operations such asaddition and subtraction are applicable. Also, in another embodiment,the drowsiness detection device 600 may change the previous referenceLF/HF ratios 6006, 6007, and 6008 to new reference LF/HF ratios 6006,6007, and 6008.

For example, the drowsiness detection device 6000 may set the referenceLF/HF ratios 6006, 6007, and 6008 on the basis of LF/HF ratios acquiredduring a certain time after the LF/HF ratio 6005 of the subject isacquired for the first time as described above. The drowsiness detectiondevice 6000 may detect a normal state on the basis of the subject'sLF/HF ratios 6005 acquired after the certain time and may change anaverage LF/HF ratio in a time interval in which the normal state isdetected to a new reference LF/HF ratio 6006, 6007, or 6008. Also, thedrowsiness detection device 6000 may change the average LF/HF ratio tothe new reference LF/HF ratio 6006, 6007, or 6008 on the basis of afirst resting LF/HF ratio, a second resting LF/HF ratio, and an activeLF/HF ratio which are acquired for the same subject after the certaintime.

As another example, the drowsiness detection device 6000 may presetreference LF/HF ratios 6006, 6007, and 6008 on the basis of the firstresting LF/HF ratio, the second resting LF/HF ratio, and the activeLF/HF ratio as described above. In this case, after the reference LF/HFratios 6006, 6007, and 6008 are set based on the first resting LF/HFratio, the second resting LF/HF ratio, and the active resting LF/HFratio, the drowsiness detection device 6000 may set new reference LF/HFratios 6006, 6007, and 6008 on the basis of a first resting LF/HF ratio,a second resting LF/HF ratio, and an active resting LF/HF ratio whichare newly set for the same subject in the same manner.

Also, the drowsiness detection device 6000 may set new reference LF/HFratios 6006, 6007, and 6008 on the basis of LF/HF ratios acquired duringa certain time after an LF/HF ratio is acquired for the first time.

Also, the drowsiness detection device 6000 may set new reference LF/HFratios 6006, 6007, and 6008 on the basis of an average LF/HF ratio in atime interval in which the subject is detected as being in the normalstate.

According to an embodiment, referring to FIG. 45, the drowsinessdetection device 6000 may detect time points t1, t2, t3, and t4 at whichthe LF/HF ratio 6005 of the subject becomes greater than (or less than)or equal to one of the plurality of reference LF/HF ratios 6006, 6007,and 6008. The detected time points may be time points for measuringlevel-specific drowsiness states and a normal state.

In an embodiment, the drowsiness detection device 6000 may detect adrowsiness state at time points t1, t2, and t3 at which the LF/HF ratio6005 of the subject becomes less than or equal to the reference LF/HFratios 6006, 6007, and 6008.

In an embodiment, at a time point t1, t2, or t3 at which the LF/HF ratio6005 of the subject becomes one of the plurality of reference LF/HFratios 6006, 6007, and 6008, the drowsiness detection device 6000 maydetect a drowsiness state with a level corresponding to the referenceLF/HF ratio. Here, the drowsiness detection device 6000 may detect adrowsiness state on the basis of a result of additionally comparing theLF/HF ratio 6005 of the subject to another reference LF/HF ratio as wellas the corresponding reference LF/HF ratio.

In detail, the plurality of reference LF/HF ratios 6006, 6007, and 6008may include a first reference LF/HF ratio 6006, a second reference LF/HFratio 6007, and a third reference LF/HF ratio 6008. For convenience ofdescription, the first reference LF/HF ratio 6006 is defined as areference LF/HF ratio for detecting the lowest-level drowsiness state.

For example, when the LF/HF ratio 6005 of the subject is less than orequal to the first reference LF/HF ratio 6006 and greater than or equalto the second reference LF/HF ratio 6007 at a first time point, thedrowsiness detection device 6000 may determine that the drowsiness stateof the subject is a first-level drowsiness state.

Also, when the LF/HF ratio 6005 of the subject is less than or equal tothe second reference LF/HF ratio 6007 and greater than or equal to thethird reference LF/HF ratio 6008 at a first time point, the drowsinessdetection device 6000 may determine that the drowsiness state of thesubject is a second-level drowsiness state.

Also, when the LF/HF ratio 6005 of the subject is less than or equal tothe third reference LF/HF ratio 6088 at a first time point, thedrowsiness detection device 6000 may determine that the drowsiness stateof the subject is a third-level drowsiness state.

A method of detecting recovery from a drowsiness state on the basis ofan LF/HF ratio will be described below with reference to FIG. 46.

According to an embodiment, referring to FIG. 46A, in the operation ofdetecting a drowsiness-related state (S6230), the drowsiness detectiondevice 6000 may detect that the subject has recovered from a drowsinessstate at time points t1, t2, t3, and t4 at which the measured LF/HFratio 6005 of the subject becomes greater than or equal to recoveryreference LF/HF ratios 6006, 6007, 6008, and 6010. Here, the drowsinessdetection device 6000 may detect recovery from the drowsiness state onthe basis of a result of additionally comparing the LF/HF ratio 6005 ofthe subject to another reference LF/HF ratio as well as thecorresponding reference LF/HF ratio.

In an embodiment, the plurality of reference LF/HF ratios 6006, 6007,6008, and 6010 may include a first recovery reference LF/HF ratio 6006,a second recovery reference LF/HF ratio 6007, and a third recoveryreference LF/HF ratio 6008. For convenience of description, the firstrecovery reference LF/HF ratio 6006 may be set as a recovery referenceLF/HF ratio which is largest among recovery reference LF/HF ratios of 1or less. For example, when the LF/HF ratio 6005 of the subject is lessthan or equal to the third recovery reference LF/HF ratio 6008 at afirst time point and is greater than or equal to the third recoveryreference LF/HF ratio 6008 and less than or equal to the second recoveryreference LF/HF ratio 6007 at a second time point subsequent to thefirst time point, the drowsiness detection device 6000 may determinethat the drowsiness state of the subject is changed from the third-leveldrowsiness state to the second-level drowsiness state at the second timepoint. Also, when the LF/HF ratio 6005 of the subject is greater than orequal to the second recovery reference LF/HF ratio 6007 and less than orequal to the first recovery reference LF/HF ratio 6006 at a third timepoint after the subject has recovered from the third-level drowsinessstate to the second-level drowsiness state, the drowsiness detectiondevice 6000 may determine that the drowsiness state of the subject ischanged from the second-level drowsiness state to the first-leveldrowsiness state at the third time point.

In an embodiment, when the LF/HF ratio 6005 of the subject is less thanor equal to the third recovery reference LF/HF ratio 6008 at a firsttime point arid is greater than or equal to the second recoveryreference LF/HF ratio 6007 and less than or equal to the first recoveryreference LF/HF ratio 6006 at a second time point subsequent to thefirst time point, the drowsiness detection device 6000 may determinethat the drowsiness state of the subject is changed from the third-leveldrowsiness state to the first-level drowsiness state at the second timepoint.

However, according to an embodiment, referring to FIG. 46B, when thedrowsiness detection device 6000 detects a time point t7 at which theLF/HF ratio 6005 of the subject is decreased to the second recoveryreference LF/HF ratio 6007 within a certain time after a time point t6at which the subject has recovered from the third-level drowsiness stateto the first-level drowsiness state, the drowsiness detection device6000 may detect that the subject is in the second-level drowsinessstate. Also, the drowsiness detection device 6000 may detect that thesubject is reset to the third-level drowsiness state.

The first recovery reference LF/HF ratio 6006 may be the same as ordifferent from the first reference LF/HF ratio 6006 shown in FIG. 45,the second recovery reference LF/HF ratio 6007 may be the same as ordifferent from the second reference LF/HF ratio 6007 shown in FIG. 45,and the third recovery reference LF/HF ratio 6008 may be the same as ordifferent from the third reference LF/HF ratio 6008 shown in FIG. 45.

Also, in order for the drowsiness detection device 6000 to detectrecovery from the drowsiness state to the normal state, a fourthrecovery reference LF/HF ratio 6010, which is greater than the first tothird recovery reference LF/HF ratios, may be set. For example, whendetecting a time point at which the LF/HF ratio 6005 of a subjectdetected as being in the drowsiness state becomes greater than or equalto the fourth recovery reference LF/HF ratio 6010, the drowsinessdetection device 6000 may determine that the subject has recovered fromthe drowsiness state to the normal state. Generally, when a subject isin the normal state, the LF/HF ratio is measured in the range of 0.9to 1. Accordingly, even if a subject is detected as being in a verylow-level drowsiness state, it may be difficult for the LF/HF of thesubject to have a value of 1 or more. That is, by setting the fourthrecovery reference LF/HF ratio 6010 greater than or equal to one as arecovery reference LF/HF ratio, the drowsiness detection device 6000 maydetermine that a subject is in the normal state only when the subjectcompletely recovers from the drowsiness state.

8.4 Heart Rate and LF/HF-Based Drowsiness Detection Method

A method of detecting drowsiness on the basis of a heart rate and anLF/HF ratio will be described below with reference to FIG. 47.

In an embodiment, the method may include level-specific drowsinessdetection operations based on a heart rate and an LF/HF ratio (S6100 andS6200) and an operation of determining a final drowsiness state (S6300).

In some embodiments, the drowsiness detection method may further includenotifying a subject or an entity other than the subject according to thelevel of the drowsiness state.

In this section, a drowsiness detection method to which the heartrate-based drowsiness detection method which has been described inSection 8.3 and the LF/HF ratio based drowsiness detection method whichhas been described in Section 8.4 are applied will be described.

The heart rate and the LF/HF ratio are due to information on heartbeatsacquired from the same subject, but in some cases, the level of thedrowsiness state determined based on the heart rate and the level of thedrowsiness state determined based on the LF/HF ratio may be differentfrom each other. For example, the drowsiness detection device may detectthe third-level drowsiness state on the basis of the heart rate anddetect the second-level drowsiness state on the basis of the LF/HF ratioin the same subject at the same time. As described above, differentdrowsy levels being determined depending on the drowsy detection methodsalthough the drowsiness detection device performs the drowsy-statedetermination on the same subject at the same time may mean that theresult of the drowsy-state determination performed by one of the drowsydetection methods may be wrong.

Therefore, in order to reduce such errors and more accurately detectdrowsiness, a drowsiness detection method based on a heart rate and anLF/HF ratio will be described below.

In an embodiment, a drowsiness-related state detected in the heart rateand LF/HF ratio-based drowsiness detection method may be expressed as afinal drowsiness state and a final normal state. In an embodiment, thefinal drowsiness state may be divided into a plurality of levelsaccording to the degree of drowsiness. For example, the final drowsinessstate may be divided into a first final drowsiness state, a second finaldrowsiness state, and a third final drowsiness state. Here, the firstfinal drowsiness state may refer to the lowest-level final drowsinessstate, and the third final drowsiness state may refer to thehighest-level final drowsiness state.

In an embodiment, the final drowsiness state and the final normal statemay refer to states corresponding to the drowsiness state and the normalstate which are detected on the basis of each of the heart rate-baseddrowsiness detection method and the LF/HF ratio-based drowsinessdetection method, respectively. That is, the final drowsiness state mayrefer to a state in which a subject is temporarily sleeping or a statein which a subject is not sleeping but is likely to sleep within apredetermined period, and the final normal state, which is not the finaldrowsiness state, may refer to a state in which a subject is unlikely tofall asleep within a predetermined time.

As described above, the drowsiness detection method based on a change inheart rate and an LF/HF ratio may be divided into a total of fourlevels, i.e., a normal state, a first-level drowsiness state, asecond-level drowsiness state, and a third-level drowsiness stateaccording to the degree of drowsiness. Here, the first-level drowsinessstate may refer to an unconscious/unaware drowsiness state, and thesecond-level drowsiness state and the third-level drowsiness state mayrefer to conscious/aware drowsiness states.

In an embodiment, the final drowsiness state divided into a plurality oflevels according to the degree of drowsiness may refer to thefirst-level drowsiness state, the second-level drowsiness state, and thethird-level drowsiness state, which are detected based on each of theheart rate-based drowsiness detection method and the LF/HF ratio-baseddrowsiness detection method. Accordingly, the first final drowsinessstate may refer to an unconscious/unaware drowsiness state. Also, thesecond final drowsiness state and the third final drowsiness state mayrefer to conscious/aware drowsiness states. In an embodiment, thedrowsiness detection device may acquire a subject's final drowsinessstate at predetermined intervals. It will be appreciated that thepredetermined interval may be fixed or variable.

In another embodiment, the drowsiness detection device may acquire asubject's final drowsiness state without following predeterminedintervals. For example, the drowsiness detection device may detect asubject's drowsiness state each time the physiological parameteracquisition device 10 or the like measures the subject's heart rateand/or LF/HF ratio, i.e., in real time.

It will be appreciated that when a subject's heart rate or LF/HF ratiois measured by the physiological parameter acquisition device 10 or thelike at predetermined intervals, the drowsiness detection device mayacquire the subject's final drowsiness state on the basis of thepredetermined intervals at which the physiological parameter acquisitiondevice 10 or the like measure the heart rate or LF/HF ratio.

As another example, the drowsiness detection device may request thephysiological parameter acquisition device 10 or the like to provide anLF/HF ratio and a heart rate upon an external input or request and mayacquire a subject's final drowsiness state each time the request ismade.

In an embodiment, the drowsiness detection device may acquire asubject's average heart rate and average LF/HF for a certain timeperiod.

The drowsiness detection device can correct noise by acquiring theaverage heart rate arid the average LF/HF ratio, and thus can have aneffect of accurately detecting drowsiness. This effect is the same as aneffect obtainable when the average heart rate or the average LF/HF ratiois acquired, and thus a detailed description thereof will be omitted.

In an embodiment, the drowsiness detection device may determine thelevel of the final drowsiness state according to the level of thedrowsiness state detected based on the heart rate and the level of thedrowsiness state detected based on the LF/HF ratio. In some cases, thelevel of the drowsiness state detected based on the heart rate and thelevel of the drowsiness state detected based on the LF/HF ratio may ormay not match each other.

For convenience of description, the normal state detected based on aheart rate is defined as a drowsiness state lower than the first-leveldrowsiness state. Also, the normal state detected based on an LF/HFratio is defined as a drowsiness state lower than the first-leveldrowsiness state.

When the level of the drowsiness state detected based on the heart rateand the level of the drowsiness state detected based on the LF/HF ratiomatch each other, the drowsiness detection device may determine thematching level as the level of the final drowsiness state. For example,when, for the subject, the drowsiness detection device detects thefirst-level drowsiness state on the basis of the heart rate and detectsthe first-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine the first-level finaldrowsiness state as the final drowsiness state. Also, when, for thesubject, the drowsiness detection device detects the second-leveldrowsiness state on the basis of the heart rate and detects thesecond-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine the second-level finaldrowsiness state as the final drowsiness state. Also, when, for thesubject, the drowsiness detection device detects the third-leveldrowsiness state on the basis of the heart rate and detects thethird-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine the third-level finaldrowsiness state as the final drowsiness state.

When the level of the drowsiness state detected based on the heart rateand the level of the drowsiness state detected based on the LF/HF ratiodo not match each other, the drowsiness detection device may determinethe final drowsiness level in consideration of various situations.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio do not match each other, the drowsiness detectiondevice may determine the higher one of the two levels as the level ofthe final drowsiness state. For example, when, for the subject, thedrowsiness detection device detects the first-level drowsiness state onthe basis of the heart rate and detects the second-level drowsinessstate on the basis of the LF/HF ratio, the drowsiness detection devicemay determine the second-level final drowsiness state as the finaldrowsiness state.

In another embodiment, when the level of the drowsiness state detectedbased on the heart rate and the level of the drowsiness state detectedbased on the LF/HF ratio do not match each other, the drowsinessdetection device may determine the lower one of the two levels as thelevel of the final drowsiness state. For example, when, for the subject,the drowsiness detection device detects the first-level drowsiness stateon the basis of the heart rate and detects the second-level drowsinessstate on the basis of the LF/HF ratio, the drowsiness detection devicemay determine the first-level final drowsiness state as the finaldrowsiness state.

In another embodiment, when the level of the drowsiness state detectedbased on the heart rate and the level of the drowsiness state detectedbased on the LF/HF ratio do not match each other, the drowsinessdetection device may determine the average of the two levels as thelevel of the final drowsiness state. Here, when the average is not aninteger, the average may be rounded up or down to the first decimalplace. For example, when, for the subject, the drowsiness detectiondevice detects the first-level drowsiness state on the basis of theheart rate and detects the third-level drowsiness state on the basis ofthe LF/HF ratio, the drowsiness detection device may determine thesecond-level final drowsiness state, which is the average of the twolevels, as the final drowsiness state. As another example, when thefirst-level drowsiness state is detected on the basis of the heart rateand the second-level drowsiness state is detected on the basis of theLF/HF ratio, the drowsiness detection device may determine thefirst-level final drowsiness state as the final drowsiness state byrounding 1.5, which is the average of the two levels, down to the firstdecimal place. Also, the drowsiness detection device may determine thesecond-level final drowsiness state as the final drowsiness state byrounding 1.5, which is the average of the two levels, up to the firstdecimal place.

It will be appreciated that, in some cases, the drowsiness detectiondevice may define a new drowsiness level between the level of thedrowsiness state detected based on the heart rate and the level of thedrowsiness state detected based on the LF/HF ratio and determine the newlevel as the final drowsiness level. For example, when the first-leveldrowsiness state is detected based on the heart rate and thesecond-level drowsiness state is detected based on the LF/HF ratio, thedrowsiness detection device may define an intermediate level drowsinessstate indicating the degree of drowsiness between the first-leveldrowsiness state and the second-level drowsiness state and determine theintermediate level drowsiness state as the final drowsiness level.

In another embodiment, when the level of the drowsiness state detectedbased on the heart rate and the level of the drowsiness state detectedbased on the LF/HF ratio do not match each other, the drowsinessdetection device may maintain the previously determined final drowsinessstate or final normal state as the drowsiness-related state. Forexample, when, for the subject, the drowsiness detection device detectsthe first-level drowsiness state on the basis of the heart rate anddetects the third-level drowsiness state on the basis of the LF/HF ratioafter detecting the first-level drowsiness state, the drowsinessdetection device may maintain the first-level final drowsiness state,which is the previously determined final drowsiness state, as thedrowsiness-related state. For example, when, for the subject, thedrowsiness detection device detects the first-level drowsiness state onthe basis of the heart rate and detects the third-level drowsiness stateon the basis of the LF/HF ratio while detecting the final normal state,the drowsiness detection device may maintain the previously determinedfinal normal state as the drowsiness-related state.

The following detailed description will focus on a case in which thelevel of the drowsiness state detected based on the heart rate and thelevel of the drowsiness state detected based on the LF/HF ratio do notmatch each other.

8.4.1 Specific Embodiment

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio match each other and both are the normal state, thedrowsiness detection device may determine that the subject is in thefinal normal state.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio match each other and both are the first-leveldrowsiness state, the drowsiness detection device may determine that thesubject is in the first-level final drowsiness state.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio match each other and both are the second-leveldrowsiness state, the drowsiness detection device may determine that thesubject is in the second-level final drowsiness state.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio match each other and both are the third-leveldrowsiness state, the drowsiness detection device may determine that thesubject is in the third-level final drowsiness state.

In an embodiment, a method of determining the lower one of the level ofthe drowsiness state detected based on the heart rate and the level ofthe drowsiness state detected based on the LF/HF ratio as the level ofthe final drowsiness state when the level of the drowsiness statedetected based on the heart rate and the level of the drowsiness statedetected based on the LF/HF ratio do not match each other (hereinafterreferred to a first method) may be applied to detect a level that needsto be less sensitively detected.

For example, a notification may be issued when the first-leveldrowsiness state, the second-level drowsiness state, and the third-leveldrowsiness state are detected. In some cases, the first-level drowsinessstate and the second-level drowsiness state may have little effect onthe safety of the subject. In this case, if the notification isfrequently given, the subject may feel uncomfortable or may not operatethe drowsiness detection device. Also, by determining the lower one ofthe level of the drowsiness state detected based on the heart rate andthe level of the drowsiness state detected based on the LF/HF ratio asthe level of the final drowsiness state, the drowsiness detection devicemay more accurately detect the drowsiness-related state. That is,determining the final drowsiness state using the drowsiness detectiondevice according to the first method may be advantageous in improvinguser convenience and accuracy.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio do not match each other, one of which indicates thenormal state, and the other of which is detected as a drowsiness statethat is one or two levels higher than the normal state, the drowsinessdetection device may determine that the subject is in the final normalstate on the basis of the normal state, which has a lower level.

For example, when the drowsiness detection device detects the normalstate on the basis of the heart rate and detects the first-leveldrowsiness state on the basis of the LF/HF ratio, the drowsinessdetection device may determine that the subject is in the final normalstate.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio do not match each other, one of which indicates thefirst-level drowsiness state, and the other of which is detected as adrowsiness state that is one level higher than the normal state, thedrowsiness detection device may determine that the subject is in thefirst-level drowsiness state on the basis of the first-level drowsinessstate, which has a lower level.

For example, when the drowsiness detection device detects thefirst-level drowsiness state on the basis of the heart rate and detectsthe second-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine that the subject is in thefirst-level drowsiness state.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio do not match each other, one of which indicates thesecond-level drowsiness state, and the other of which is detected as adrowsiness state that is higher than the normal state, the drowsinessdetection device may determine that the subject is in the second-leveldrowsiness state on the basis of the second-level drowsiness state,which has a lower level. On the other hand, a method of determining thehigher one of the level of the drowsiness state detected based on theheart rate and the level of the drowsiness state detected based on theLF/HF ratio as the level of the final drowsiness state when the level ofthe drowsiness state detected based on the heart rate and the level ofthe drowsiness state detected based on the LF/HF ratio do not match eachother (hereinafter referred to a second method) may be applied to detecta level that needs to be more sensitively detected.

For example, a notification may be issued when the first-leveldrowsiness state, the second-level drowsiness state, and the third-leveldrowsiness state are detected. In some cases, the third-level drowsinessstate is the highest degree of drowsiness state and may directly affectthe safety of the subject. In this case, when the drowsiness detectiondevice does not give a strong notification to the subject as soon as thethird-level drowsiness state is detected, the subject may be in aseriously dangerous situation. Also, by determining the higher one ofthe level of the drowsiness state detected based on the heart rate andthe level of the drowsiness state detected based on the LF/HF ratio asthe level of the final drowsiness state, the drowsiness detection devicemay determine a level to be detected in various situations as the levelof the final drowsiness state. That is, determining the final drowsinessstate using the drowsiness detection device according to the secondmethod may be advantageous in reducing user risk.

In an embodiment, when the level of the drowsiness state detected basedon the heart rate and the level of the drowsiness state detected basedon the LF/HF ratio do not match each other, one of which indicates thethird-level drowsiness state, and the other of which is detected as adrowsiness state that is one, two, or three levels lower than thethird-level drowsiness state, the drowsiness detection device maydetermine that the subject is in the third-level final drowsiness stateon the basis of the third-level drowsiness state, which has a higherlevel.

For example, when the drowsiness detection device detects thethird-level drowsiness state on the basis of the heart rate and detectsthe first-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine that the subject is in thethird-level drowsiness state.

In another embodiment, the drowsiness detection device may follow thesecond method in order to detect the second final drowsiness state. Thisis because in some cases, the drowsiness detection device may need todetect not only the third-level drowsiness state but also thesecond-level drowsiness state in more various situations. In detail,when the subject is a freight vehicle driver or a public transportvehicle driver and is in danger of a traffic accident, the drivers andpassengers of nearby vehicles may be in serious danger in addition tothe driver. Accordingly, when the level of the drowsiness state detectedbased on the heart rate and the level of the drowsiness state detectedbased on the LF/HF ratio do not match each other, one of which indicatesthe second-level drowsiness state, and the other of which is detected asa drowsiness state that is one or two levels lower than the second-leveldrowsiness state, the drowsiness detection device may determine that thesubject is in the second-level final drowsiness state on the basis ofthe second-level drowsiness state, which has a higher level.

For example, when the drowsiness detection device detects thesecond-level drowsiness state on the basis of the heart rate and detectsthe first-level drowsiness state on the basis of the LF/HF ratio, thedrowsiness detection device may determine that the subject is in thesecond-level drowsiness state.

9. Various Applications Using Physiological Parameter Acquisition Device9.1 Various Embodiments of Display Device

The above-described physiological parameter acquisition method ordrowsiness detection method may be used in various applications. Forexample, the physiological parameter acquisition method or drowsinessdetection method may be used in a display device.

In this case, the display device may include a smart mirror including amirror display, but the present invention is not limited thereto. Thedisplay device may refer to a device with a display, such as asmartphone, a tablet, and the like.

Also, a physiological parameter or physiological information acquiredaccording to the physiological parameter acquisition method may beoutput through the display, and also drowsiness information acquiredaccording to the drowsiness detection method may be output through thedisplay.

Also, the display device may perform an operation corresponding to thephysiological parameter of physiological information acquired accordingto the physiological parameter acquisition method.

Also, the display device may perform an operation corresponding to thedrowsiness information acquired according to the drowsiness detectionmethod.

Also, a user of the display device may set a physiological parameter ofinterest, and when a physiological parameter of interest is set, onlythe physiological parameter of interest may be acquired or output. Forexample, when a first user sets a heart rate and a second user sets ablood pressure, the first user may acquire only a heart rate, and thesecond user may acquire only a blood pressure, but the present inventionis not limited thereto.

Also, the display device may be installed in a public place such as ahotel lobby, a hotel front desk, and a public restroom and used tomeasure physiological parameters of incoming or outgoing people, but thepresent invention is not limited thereto.

Also, the operation of the physiological parameter measurement devicewill be described below, but the operation to be described may beperformed even by other processors such as an electronic control unit(ECU) mounted on a vehicle or may be performed even by a processorincluded in a server.

Also, a device that will be described as the physiological parametermeasurement device may refer to an image acquisition device. In thiscase, the operation to be described may be performed by other processorssuch as an ECU or a processor included in a server.

FIG. 48 is a diagram illustrating a smart mirror device according to anembodiment.

Referring to FIG. 48, a smart mirror device 7000 according to anembodiment may include at least one of an image sensor 7010, a mirrordisplay 7020, and a control unit 7030. For example, the smart mirrordevice 7000 may include the mirror display 7020 and the control unit7030, but the present invention is not limited thereto.

In this case, the image sensor 7010 may be provided as a visible camerafor acquiring a visible light image, an infrared (IR) camera foracquiring an infrared image, and the like. However, the presentinvention is not limited thereto, and a hybrid-type camera for acquiringa visible light image and an infrared image may be provided.

Also, the image sensor 7010 may be provided as one package with themirror display 7020. However, the present invention is not limitedthereto, and the image sensor 7010 may be provided as a separate unitdistinct from the mirror display 7020.

Also, the image sensor 7010 may acquire a plurality of image frames andtransmit an acquired image frame to the control unit 7030.

Also, the mirror display 7020 may refer to a means for forwardinginformation while functioning as a looking glass, but the presentinvention is not limited thereto.

Also, the mirror display 7020 may include a mirror configured tofunction as a looking glass and a display configured to forwardinformation and may be provided in a form in which a mirror film isadded to the display, but the present invention is not limited thereto.

Also, the mirror display 7020 may include a half mirror that may referto a translucent looking glass configured to transmit light in onedirection but reflect light in another direction, but the presentinvention is not limited thereto.

Also, the mirror display 7020 may include a polarizing plate, but thepresent invention is not limited thereto.

Also, the mirror display 7020 may include a mirror display that can becommonly understood in addition to the above-described exemplaryprinciples and exemplary elements. In detail, the mirror display 7020may be understood as a device configured to forward information whilefunctioning as a mirror.

Also, the control unit 7030 may acquire a physiological parameter andphysiological information on the basis of a plurality of image framesacquired from the image sensor 7010.

In this case, the above description is applicable to a method ofacquiring a physiological parameter and physiological information on thebasis of a plurality of image frames acquired from the image sensor7010, and thus a redundant description thereof will be omitted.

Also, the control unit 7030 may acquire a physiological parameter andphysiological information from an external sensor (not shown). Forexample, the control unit 7030 may acquire heartbeat information throughan ECG sensor attached to a subject or the like, but the presentinvention is not limited thereto.

Also, the control unit 7030 may acquire personal statistical datathrough an input device (not shown). For example, the control unit 7030may acquire personal statistical data such as a subject's height, age,and weight through a keyboard, but the present invention is not limitedthereto.

In this case, the input device may be a keyboard, a mouse, a keypad, adome switch, a touchpad (e.g., a static pressure/capacitance), a jogwheel, or a jog switch, but the present invention is not limitedthereto.

Also, the control unit 7030 may acquire personal statistical datathrough an external device (not shown). For example, the control unit7030 may acquire personal statistical data such as a subject's height,age, and weight through the subject smartphone, but the presentinvention is not limited thereto.

In this case, the external device may include a mobile terminal, such asa cellular phone, a smartphone, a laptop computer, a digitalbroadcasting terminal, a personal digital assistant (PDA), portablemultimedia player (PMP), and a navigation device, and also a stationaryterminal, such as a digital TV and a desktop computer, but the presentinvention is not limited thereto.

Also, the control unit 7030 may control the operation of the mirrordisplay 7020. For example, the control unit 7030 may control theoperation of the mirror display 7020 so that the physiological parameteror physiological information acquired based on the image frames isoutput, but the present invention is not limited thereto.

Also, the control unit 7030 may control the operation of the mirrordisplay 7020 so that various pieces of information are output. Forexample, the control unit 7030 may control the operation of the mirrordisplay 7020 so that various pieces of information such as weatherinformation, date information, calendar information, internal humidityinformation, and internal temperature information are output, but thepresent invention is not limited thereto.

Also, the control unit 7030 may control the operation of the mirrordisplay 7020 so that various pieces of personal information are output.For example, the control unit 7030 may control the operation of themirror display 7020 so that various pieces of personal information suchas a subject's schedule information and medication information areoutput, but the present invention is not limited thereto.

FIG. 49 is a diagram illustrating a smart mirror device according to anembodiment.

Referring to FIG. 49, a smart mirror device 7100 according to anembodiment may include an image sensor 7110 and a mirror display 7120.

In this case, the above-described operations are applicable to the imagesensor 7110 and the mirror display 7120, and thus a redundantdescription thereof will be omitted.

According to an embodiment, basic information may be output to themirror display 7120. For example, as shown in FIG. 49, at least onepiece of the basic information such as date information, timeinformation, external temperature information, weather information,calendar information, and news information may be output, but thepresent invention is not limited thereto.

Also, personal information may be output to the mirror display 7120. Forexample, as shown in FIG. 49, at least one piece of the personalinformation such as a subject's age, height, and weight may be output tothe mirror display 7120, but the present invention is not limitedthereto.

Also, for example, as shown in FIG. 49, at least one piece of thepersonal information such as a subject's schedule information andmedication information may be output to the mirror display 7120, but thepresent invention is not limited thereto.

Also, a subject's physiological parameters may be output to the mirrordisplay 7120. For example, as shown in FIG. 49, at least one of thephysiological parameters, such as a heart rate, an oxygen saturationlevel, and a blood pressure, may be output to the mirror display 7120,but the present invention is not limited thereto.

In this case, the physiological parameter may be acquired based on animage frame acquired from the image sensor 7110. However, the presentinvention is not limited thereto, and the physiological parameter may beacquired by an external sensor or the like.

Also, a subject's physiological information may be output to the mirrordisplay 7120. For example, as shown in FIG. 49, at least one piece ofthe physiological information such as condition information may beoutput to the mirror display 7120, but the present invention is notlimited thereto.

In this case, the physiological information may be acquired based on animage frame acquired from the image sensor 7110. However, the presentinvention is not limited thereto, and the physiological information maybe acquired by an external sensor or the like.

Also, the physiological information may be acquired based on at leastone piece of the physiological information, but the present invention isnot limited thereto.

Also, a subject's physiological signals may be output to the mirrordisplay 7120. For example, as shown in FIG. 49, at least one of thephysiological signals such as a heartbeat signal may be output to themirror display 7120, but the present invention is not limited thereto.

In this case, the physiological signal may be acquired based on an imageframe acquired from the image sensor 7110. However, the presentinvention is not limited thereto, and the physiological signal may beacquired based on an external sensor or the like.

Also, the mirror display 7120 may include an input device. For example,as shown in FIG. 49, the mirror display 7120 may include at least oneinput device such as a touch panel, but the present invention is notlimited thereto.

Also, for example, although not shown in FIG. 49, the smart mirrordevice 7100 may track a user's pupil or the like or recognize a user'sgesture to receive an input from the user, but the present invention isnot limited thereto.

Also, the mirror display 7120 may acquire information regarding asubject through the input device, and the acquired information regardingthe subject may be output to the mirror display 7120.

Also, the above drawing and description of the mirror display 7120 aremerely an example, and it is obvious that various pieces of informationmay be output in various ways without being limited to FIG. 49 and arelated description.

9.1.1 Various Embodiments of Display Device in which Guide Region isOutput

FIG. 50 is a diagram illustrating a smart mirror device in which a guideregion is output according to an embodiment.

Referring to FIG. 50, a smart mirror device 7150 according to anembodiment may include an image sensor 7160 and a mirror display 7170.The above description is applicable to the image sensor 7160 and themirror display 7170, and thus a redundant description thereof will beomitted.

Also, a guide region 7180 may be displayed in the mirror display 7170according to an embodiment. In this case, the guide region 7180 mayfunction to approximate a measurement position of a subject.

When a physiological parameter is acquired using an image sensor,measurement accuracy may vary depending on the measurement position ofthe subject. Thus, when the guide region 7180 for guiding themeasurement position of the subject is utilized, it is possible toimprove the accuracy of the measurement of the subject's physiologicalparameter.

Also, the guide region 7180 may be displayed in a rectangular shape asshown in FIG. 50. The present invention is not limited thereto, and theguide region 7180 may be displayed in various shapes such as a humanface contour, a circle, and an ellipse.

Also, a position at which the guide region 7180 is displayed may varydepending on the subject. For example, a guide region 7180 for a tallsubject may be relatively positioned in an upper portion of the mirrordisplay 7108, and a guide region 7180 for a short subject is relativelypositioned in a lower portion of the mirror display 7170, but thepresent invention is not limited thereto.

Also, the position of the guide region 7180 may be changed in real time.For example, when a subject moves during measurement, the position ofthe guide region 7180 may be changed in real time in response to themovement of the subject, but the present invention is not limitedthereto.

Also, the guide region 7180 may be displayed before a subject'sphysiological parameter is measured. For example, before a subjectenters a measurement region, the guide region 7180 may be displayed tonotify the subject of an approximate measurement position, but thepresent invention is not limited thereto.

Also, when there are a plurality of people to be measured, the guideregion 7180 may function to provide a notification about a subject to bemeasured. For example, when person A and person B enter a measurementregion and only person B is a subject, the guide region 7180 may bedisplayed to correspond to person B, but the present invention is notlimited thereto.

9.1.2 Various Embodiments of Display Device in which PredeterminedInformation is Displayed During Physiological Parameter Measurement

When a physiological parameter is acquired using an image sensor,measurement accuracy may vary depending on subject movement, and thus afunction for minimizing subject movement may be required duringphysiological parameter measurement.

Accordingly, when a display device in which predetermined information isdisplayed during physiological parameter measurement is used, it ispossible to induce subject movement to be minimized using the displayedinformation.

FIG. 51 is a diagram illustrating a smart mirror device in whichpredetermined information is output according to an embodiment.

Referring to FIG. 51, depending on the physiological parametermeasurement time, different information may be displayed in the smartmirror device 7200 according to an embodiment.

In detail, physiological parameter measurement for a subject may bestarted at a first time point 7210.

In this case, the first time point 7210 may be a time point at which ameasurement target region of the subject is positioned in a measurementregion. However, the present invention is not limited thereto, and thefirst time point 7210 may be a time point at which a person to which themeasurement is to be performed is positioned within the angle of view ofan image sensor.

Also, the first time point 7210 may be a time point at which a subjectenters an intention for measurement. For example, the first time point7210 may be a time point at which the subject touches a measurementbutton, but the present invention is not limited thereto.

Also, first information may be displayed at a second time point 7220.

In this case, the second time point 7220 may be a time point at which afacial recognition for the subject is completed, but the presentinvention is not limited thereto.

Also, the second time point 7220 may refer to a predetermined time afterthe physiological parameter measurement for the subject is started, butthe present invention is not limited thereto.

Also, the first information may be recognition information of thesubject. For example, as shown in FIG. 51, the first informationdisplayed at the second tune point 7220 may be greeting information ofthe subject whose face has been recognized, but the present invention isnot limited thereto.

Also, the first information may be information regarding a measurementprocessing time. For example, when the second tune point 7220 is twoseconds after the measurement is started, information corresponding totwo seconds may be displayed.

Also, the first information may be information regarding the timeremaining until the end of the measurement. For example, it takes sixseconds to measure a physiological parameter. When the second time point7220 is two seconds after the measurement is started, informationcorresponding to the remaining four seconds may be displayed.

Also, second information may be displayed at a third time point 7230.

In this case, the third time point 7230 may refer to a predeterminedtime after the facial recognition for the subject is completed, but thepresent invention is not limited thereto.

Also, the third time point 7230 may refer to a predetermined time afterthe physiological parameter measurement for the subject is started, butthe present invention is not limited thereto.

Also, the second information may be health information of the subject.For example, as shown in FIG. 51, the second information displayed atthe third time point 7230 may be a monthly average physiologicalparameter of the subject, but the present invention is not limitedthereto.

Also, the second information may be information regarding a measurementprocessing time. For example, when the third time point 7230 is fourseconds after the measurement is started, information corresponding tofour seconds may be displayed.

Also, the second information may be information regarding the timeremaining until the end of the measurement. For example, it takes sixseconds to measure a physiological parameter. When the third time point7230 is four seconds after the measurement is started, informationcorresponding to the remaining two seconds may be displayed.

Also, third information may be displayed at a fourth time point 7240.

In this case, the fourth time point 7240 may be a time point at whichthe physiological parameter measurement for the subject is completed,but the present invention is not limited thereto.

Also, the third information may be information related to aphysiological parameter of the subject. For example, as shown in FIG.51, the third information displayed at the fourth time point 7240 may bea heart rate, an oxygen saturation level, and a blood pressure of thesubject, but the present invention is not limited thereto.

Also, the third information may be information related to physiologicalinformation of the subject. For example, although not shown in FIG. 51,the fourth information displayed at the fourth time point 7240 may bephysiological information such as a condition index of the subject, butthe present invention is not limited thereto.

9.1.3 Various Embodiments of Smart Mirror Device in which Image AcquiredThrough Image Sensor is Displayed

FIG. 52 is a diagram illustrating a smart mirror device according to anembodiment.

Referring to FIG. 52, a smart mirror device 7250 according to anembodiment may include an image sensor 7260 and a mirror display 7270.

In this case, the above-described operations are applicable to the imagesensor 7260 and the mirror display 7270, and thus a redundantdescription thereof will be omitted.

According to an embodiment, an image may be displayed on the mirrordisplay 7270. For example, as shown in FIG. 52, an image acquiredthrough the image sensor 7260 may be displayed on the mirror display7270, but the present invention is not limited thereto.

Also, information regarding a subject to be measured may be shown in thedisplayed image. For example, a region included in the image to displaya measurement target may be showed, but the present invention is notlimited thereto.

As described above, by additionally displaying the image even though thesubject can see himself or herself through the mirror display 7270, asubject whose physiological parameter is to be measured may bedefinitely shown when there are a plurality of measurement targets, andthus it is possible to prevent confusion among a plurality of people whomay be the measurement targets.

Also, the displayed image of the subject and the shape of the subjectreflected through the mirror display 7270 may be positioned at differentplaces. However, the present invention is not limited thereto, and thedisplayed image of the subject and the shape of the subject may overlapeach other.

9.1.4 Various Embodiments of Display Device Configured to MeasurePhysiological Parameter in Real Time

FIG. 53 is a diagram illustrating a display device configured to measurea physiological parameter in real time according to an embodiment.

Referring to FIG. 53, a display device 7300 according to an embodimentmay measure a physiological parameter in real time, and a physiologicalparameter may be displayed on the display device 7300 in real time.

In detail, a physiological parameter of a subject acquired in a firsttime period may be displayed at a first time point 7310.

In this case, the first time point 7310 may be a time point after thephysiological parameter of the subject is acquired in the first timeperiod.

In detail, a physiological parameter of a subject acquired in a secondtime period may be displayed at a second time point 7320.

In this case, the second time point 7320 may be a time point after thephysiological parameter of the subject is acquired in the second timeperiod.

Also, the second time period and the first time period may differ fromeach other and may at least partially overlap each other.

Also, the second time point 7320 may be later than the first time point7310.

Also, a physiological parameter of a subject acquired in a third timeperiod may be displayed at a third time point 7330.

In this case, the third time point 7330 may be a time point after thephysiological parameter of the subject is acquired in the third timeperiod.

Also, the third time period may differ from the first and second timeperiods and may at least partially overlap the first and second timeperiods.

Also, the third time point 7330 may be later than the first time point7310 and the second time point 7320.

Also, a physiological parameter of a subject acquired in a fourth timeperiod may be displayed at a fourth time point 7340.

In this case, the fourth time point 7340 may be a time point after thephysiological parameter of the subject is acquired in the fourth timeperiod.

Also, the fourth time period may differ from the first, second, andthird time periods and may at least partially overlap the first, second,and third time periods.

Also, the fourth time point 7340 may be later than the first, second,and third time points 7310, 7320, and 7330.

Also, as described above, when a physiological parameter is acquired inreal time, a physiological parameter acquired at a specific time pointmay be used to store a final physiological parameter. For example, aphysiological parameter acquired for the first time may be stored, butthe present invention is not limited thereto. A physiological parameteracquired for the last time may be stored, and a physiological parameteracquired during physiological parameter measurement may be stored.

Also, as described above, when a physiological parameter is acquired inreal time, a plurality of physiological parameters may be used to storea final physiological parameter. For example, the average of a pluralityof physiological parameters acquired during a certain time period may bestored as a final physiological parameter, but the present invention isnot limited thereto.

Also, as described above, when a physiological parameter is acquired inreal time, a plurality of physiological parameters are acquired evenwhen noise due to subject movement occurs during the physiologicalparameter measurement. Thus, it is possible to acquire a more accuratephysiological parameter.

Also, as described above, when a physiological parameter is acquired inreal time, an update period may be set to display the physiologicalparameter. For example, a user may set an update period to display thephysiological parameter, and the physiological parameter may be updatedaccording to the set period, but the present invention is not limitedthereto.

9.1.5 Various Embodiments of Display Device in which PredeterminedInformation is Displayed During Physiological Parameter Measurement

FIG. 54 is a diagram illustrating a smart mirror device in whichpredetermined information is output according to an embodiment.

Referring to FIG. 54, depending on the physiological parametermeasurement time, different information may be displayed in the smartmirror device 7350 according to an embodiment.

In detail, a first time point 7360 may be a time point at which orbefore physiological parameter measurement for a subject is started.

In this case, the first time point 7360 may be a time point at which ameasurement target region of the subject is positioned in a measurementregion. However, the present invention is not limited thereto, and thefirst time point 7360 may be a time point at which a person, on whichthe measurement is to be performed, is positioned within the angle ofview of an image sensor.

Also, first information may be displayed at the first time point 7360.

In this case, the first information may include basic information. Forexample, as shown in FIG. 54, the first information may include basicinformation such as date information, time information, and weatherinformation, but the present invention is not limited thereto.

Also, the first information may include information regarding a guideregion. For example, as shown in FIG. 54, a guide region may bedisplayed at the first time point 7360, but the present invention is notlimited thereto.

Also, second information may be displayed at a second time point 7370.

In this case, the second time point 7370 may be a time point at whichthe physiological parameter of the subject is being measured, but thepresent invention is not limited thereto.

Also, the second time point 7370 may refer to a predetermined time afterthe physiological parameter measurement for the subject is started, butthe present invention is not limited thereto.

Also, the second information may be personal information of the subject.For example, as shown in FIG. 54, the second information displayed atthe second time point 7370 may include personal information, such as thesubject's main schedule items and medication information, but thepresent invention is not limited thereto.

Also, the second information may include information related tophysiological parameter measurement. For example, as shown in FIG. 54,the second information displayed at the second time point 7370 mayinclude information indicating that the subject's physiologicalparameter is being measured, but the present invention is not limitedthereto.

Also, third information may be displayed at a third time point 7380.

In this case, the third time point 7380 may be a time point at which thephysiological parameter measurement for the subject is completed.However, the present invention is not limited thereto, and whenmeasurement is made in real time, the third time point 7380 may be atime point at which the measurement is completed at least once.

Also, the third information may include the measured physiologicalparameter. For example, as shown in FIG. 54, the third informationdisplayed at the third time point 7380 may include a measured heartrate, oxygen saturation level, and blood pressure, but the presentinvention is not limited thereto.

Also, as shown in FIG. 54, the first information and the secondinformation may be simultaneously displayed at the second time point7370, and the first information, the second information, and the thirdinformation may be simultaneously displayed at the third time point7380.

9.1.6 Various Embodiments of Smart Mirror Device Including SwitchingDevice

FIG. 55 is a diagram illustrating a smart mirror device including aswitching device according to an embodiment.

Referring to FIG. 55, a smart mirror device 7400 according to anembodiment may include an image sensor 7410, a mirror display 7420, anda switching device 7430.

In this case, the above description is applicable to the image sensor7410 and the mirror display 7420, and thus a redundant descriptionthereof will be omitted.

Also, herein, an open state of the switching device 7430 may refer to astate in which a window for the image sensor 7410 acquiring an image isobtained, and a closed state of the switching device 7430 may refer to astate in which a window for the image sensor 7410 acquiring an image isnot obtained.

Also, the image sensor 7410 may be used to detect the open state and theclosed state. For example, the closed state may be detected whenillumination detected by the image sensor 7410 is less than or equal toa reference value, and the open state may be detected when theillumination is greater than or equal to the reference value. However,the present invention is not limited thereto, and the open state and theclosed state may be detected using the image sensor 7410 in variousways.

Also, an external sensor may be used to detect the open state and theclosed state. For example, the open state may be detected when theexternal sensor detects the degree to which the switching device 7430 isopen, and the closed state may be detected when the external sensordetects the degree to which the switching device 7430 is closed.However, the present invention is not limited thereto, and the openstate and the closed state may be detected using the external sensor invarious ways.

According to an embodiment, the switching device 7430 may open or closethe image sensor 7410. For example, the switching device 7430 may openor close the front of the image sensor 7410 housed in an internalhousing space to adjust an image acquired by the image sensor 7410 orprevent the image sensor 7410 from being visible from the outside.

In detail, at a first time point 7440 at which the switching device 7430is positioned in the closed state, the image sensor 7410 may not bevisible from the outside due to the switching device 7430.

Also, the image sensor 7410 may acquire no image at the first time point7440. For example, the image sensor 7410 may be powered off through theclosing operation of the switching device 7430, but the presentinvention is not limited thereto.

Also, at a second time point at which the switching device 7430 ispositioned in the open state, the image sensor 7410 may be visible fromthe outside.

Also, the image sensor 7410 may acquire an image at the second timepoint 7450. For example, the image sensor 7410 may be powered on throughthe opening operation of the switching device 7430, but the presentinvention is not limited thereto.

Also, the surface of the switching device 7430 may be formed as a mirrorand may be formed of the same material as that of the external surfaceof the mirror display 7420, but the present invention is not limitedthereto.

Also, the switching device 7430 may function to protect the image sensor7410 from external dust or the like. For example, when the switchingdevice 7430 is in the closed state, it is possible to prevent externaldust or the like from coming into contact with the image sensor 7410,but the present invention is not limited thereto.

9.1.7 Various Embodiments of Smart Mirror Device Placed on Shoe Rack

A smart mirror device placed above a shoe rack of a house or at theentrance of a building may measure physiological parameters for peoplewho come in or go out. Usually, a shoe rack of a house or the entranceof a building is a place where people wear clothes, and has few privacyissues when an image sensor is used. Thus, it may be easy to install animage sensor for measuring a physiological parameter.

A smart mirror device placed at the entrance of a building or the like,which may be represented with a shoe rack, will be described below, andfor convenience of description, a smart mirror device placed above ashoe rack will be described.

FIG. 56 is a diagram illustrating a smart mirror device placed above ashoe rack according to an embodiment.

Referring to FIG. 56, a smart mirror device 7460 according to anembodiment may include an image sensor 7470 and a mirror display 7480.

In this case, the above description is applicable to the image sensor7470 and the mirror display 7480, and thus a redundant descriptionthereof will be omitted.

The image sensor 7470 according to an embodiment may acquire an image ofa subject 7490. For example, the image sensor 7470 may acquire an imageof a subject 7490 who passes by a shoe rock on which the smart mirrordevice 7460 is placed

Also, the smart mirror device 7460 may acquire a physiological parameteron the basis of an image frame acquired from the image sensor 7470.

Also, the mirror display 7480 may reflect and provide the shape of thesubject 7490. For example, the subject 7490 may observe himself orherself through the mirror display.

Also, the mirror display 7480 may display a physiological parameter ofthe subject 7490. For example, the mirror display 7480 may display aphysiological parameter acquired based on an image frame acquired fromthe image sensor 7470, but the present invention is not limited thereto.

Also, the mirror display 7480 may display physiological information ofthe subject 7490. For example, the mirror display 7480 may displayphysiological information acquired based on the acquired physiologicalparameter, but the present invention is not limited thereto.

Also, the mirror display 7480 may be formed on at least a portion of thesmart mirror device 7460. For example, as shown in FIG. 56, the mirrordisplay 7480 may be placed in a central part of the smart mirror device7460 to occupy a certain area.

Also, although not shown in FIG. 56, it is obvious that the mirrordisplay 7480 may display basic information, personal information, andthe like.

9.1.7.1 Various Embodiments of Smart Mirror Device Using Trigger Signal

When a smart mirror device for measuring a physiological parameteroperates continuously, power may continue to be consumed in order toacquire and analyze an image. Accordingly, a trigger signal may be usedto provide a smart mirror device capable of saving energy and operatingmore smartly.

FIG. 57 is a flowchart illustrating a smart mirror device operatingmethod according to an embodiment.

Referring to FIG. 57, a smart mirror device operating method 7500according to an embodiment may include at least one of an operation ofacquiring an on-trigger (S7510), an operation of acquiring aphysiological parameter of a subject (S7520), an operation of acquiringan off-trigger (S7530), and an operation of stopping at least oneoperation of a smart mirror device (S7540), but the present invention isnot limited thereto.

In detail, the smart mirror device operating method 7500 according to anembodiment may include an operation of acquiring the on-trigger (S7510).

In this case, the on-trigger may be a trigger signal for starting atleast one operation of the smart mirror device. For example, theon-trigger may be a trigger signal for performing an operation of animage sensor acquiring an image. However, the present invention is notlimited thereto, and the on-trigger may be a trigger signal for startingat least one operation of the smart mirror device, e.g., an operation ofpowering the smart mirror device on.

Also, the on-trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detectionsensor, the on-trigger may be acquired from the motion detection sensor.In detail, when a predetermined motion is detected by the motiondetection sensor, the smart mirror device may acquire a correspondingtrigger signal, and the trigger signal may become the on-trigger, butthe present invention is not limited thereto.

Also, in this case, the motion detection sensor may be placed inside thesmart mirror device. However, the present invention is not limitedthereto, and the motion detection sensor may be placed. outside thesmart mirror device.

Also, for example, when the smart mirror device includes an input devicesuch as a touch panel, the on-trigger may be acquired from the inputdevice. In detail, when a predetermined input is acquired from the inputdevice, the smart mirror device may acquire a corresponding triggersignal, and the trigger signal may become the on-trigger, but thepresent invention is not limited thereto.

Also, for example, when the smart mirror device includes an imagesensor, the on-trigger may be acquired from the image sensor. In detail,a lighting device placed above a shoe rack detects a predeterminedmotion and emits light, and the image sensor may detect light emittedfrom the lighting device. In this case, the smart mirror device mayacquire a corresponding trigger signal, and the trigger signal maybecome the on-trigger, but the present invention is not limited thereto.

Also, for example, when an entrance door is included within the angle ofview of the image sensor, the on-trigger may be acquired based on achange in the entrance door. In detail, the image sensor may determinethat the entrance door is open, the smart mirror device may acquire acorresponding trigger signal, and the trigger signal may become theon-trigger, but the present invention is not limited thereto.

Also, the smart mirror device operating method 7500 according to anembodiment may include an operation of acquiring a physiologicalparameter of a subject (S7520).

In this case, a physiological signal for the subject may be acquiredbased on an image frame acquired through the image sensor. However, thishas been described above, and thus a redundant description thereof willbe omitted.

Also, the smart mirror device operating method 7500 according to anembodiment may include acquiring the off-trigger (S7530).

In this case, the off-trigger may be a trigger signal for stopping atleast one operation of the smart mirror device. For example, theoff-trigger may be a trigger signal for stopping an operation of animage sensor acquiring an image. However, the present invention is notlimited thereto, and the off-trigger may be a trigger signal forstopping at least one operation of the smart mirror device, e.g., anoperation of powering the smart mirror device off.

Also, the off-trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detectionsensor, the off-trigger may be acquired from the motion detectionsensor. In detail, when a predetermined motion has not been detected bythe motion detection sensor during a certain time, the smart mirrordevice may acquire a corresponding trigger signal, and the triggersignal may be the off-trigger, but the present invention is not limitedthereto.

Also, for example, when the smart mirror device includes an input devicesuch as a touch panel, the off-trigger may be acquired from the inputdevice. In detail, when a predetermined input is acquired from the inputdevice, the smart mirror device may acquire a corresponding triggersignal, and the trigger signal may become the off-trigger, but thepresent invention is not limited thereto.

Also, for example, when the smart mirror device includes an imagesensor, the off-trigger may be acquired from the image sensor. Indetail, a lighting device placed above a shoe rack has not detected apredetermined motion during a certain time and is turned off. The imagesensor may detect the intensity of light to determine that the lightdevice is turned off, the smart mirror device may acquire acorresponding trigger signal, and the trigger signal may become theoff-trigger, but the present invention is not limited thereto.

Also, for example, the off-trigger may be acquired based on theacquisition of a physiological parameter by the smart mirror device. Indetail, when a measurement target region is not present within the angleof view of the image sensor, no physiological parameter may be acquired.When no physiological parameter is detected during a certain time, thesmart mirror device may acquire a corresponding trigger signal, and thetrigger signal may become the off-trigger, but the present invention isnot limited thereto.

Also, for example, when an entrance door is included within the angle ofview of the image sensor, the off-trigger may be acquired based on achange in the entrance door. In detail, the image sensor may determinethat the entrance door is open, the smart mirror device may acquire acorresponding trigger signal, and the trigger signal may become theoff-trigger, but the present invention is not limited thereto.

FIGS. 58 and 59 are diagrams illustrating an operation of a smart mirrordevice using a trigger signal according to an embodiment.

Referring to FIGS. 58 and 59, a smart mirror device 7550 according to anembodiment may include an image sensor 7560 and a mirror display 7570.

In this case, the above description is applicable to the image sensor7560 and the mirror display 7570, and thus a redundant descriptionthereof will be omitted.

According to an embodiment, referring to FIG. 58, when a subject 7580 ispositioned in a predetermined on-trigger occurrence region, at least oneoperation of the smart mirror device may be started.

In this case, when the smart mirror device includes a motion detectionsensor, the predetermined on-trigger occurrence region may be thedetection range of the motion detection sensor and may be the detectionrange of the motion detection sensor of a lighting device installed on ashoe rack, but the present invention is not limited thereto.

Also, according to an embodiment, referring to FIG. 59, when the smartmirror device 7550 receives a predetermined input from the subject 7580,at least one operation of the smart mirror device 7550 may be started.

In this case, the predetermined input may be acquired through an inputdevice such as a touch panel, but the present invention is not limitedthereto.

9.1.7.2 Various Embodiments of Smart Mirror Device Configured to MeasurePhysiological Parameter According to Priority

When a plurality of people are present in a physiological parametermeasurement region of a smart mirror device, a physiological parametermay be measured according to priority.

FIG. 60 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment.

Referring to FIG. 60, a smart mirror device operating method 7600according to an embodiment may include at least one of an operation ofacquiring an image including a plurality of people (S7610), an operationof determining priority for physiological parameter measurement (S7620),an operation of acquiring a physiological parameter according to thepriority (S7630), and an operation of outputting the acquiredphysiological parameter (S7640), but the present invention is notlimited thereto.

In detail, the smart mirror device operating method 7600 according to anembodiment may include an operation of acquiring an image including aplurality of people (S7610). The above description about imageacquisition is applicable to this operation, and thus a redundantdescription thereof will be omitted.

Also, the smart mirror device operating method 7600 according to anembodiment may include an operation of determining priority forphysiological parameter measurement (S7620) and an operation ofacquiring a physiological parameter according to the priority (S7630).

In this case, the ease of the physiological parameter measurement may beconsidered to determine the priority. For example, a physiologicalparameter of a person who occupies the largest area in the acquiredimage among the plurality of people may be measured, but the presentinvention is not limited thereto.

Also, for example, a physiological parameter of a person who is actuallyrecognized as a person in the acquired image among the plurality ofpeople may be measured, but the present invention is not limitedthereto.

Also, for example, a physiological parameter of a person who occupies anarea closet to the center in the acquired image among the plurality ofpeople may be measured, but the present invention is not limitedthereto.

Also, for example, a physiological parameter of a person who ispositioned closest to the image sensor in the acquired image among theplurality of people may be measured, but the present invention is notlimited thereto.

Also, prestored information may be utilized to determine the priority.For example, a physiological parameter of a person who is stored as asubject among the plurality of people may be measured, but the presentinvention is not limited thereto.

Also, for example, a physiological parameter may be acquired accordingto preset priority. In detail, when a child among family members is setto have high priority and a child and his or her father is positioned inthe physiological parameter measurement region, a physiologicalparameter of the child may be measured, but the present invention is notlimited thereto.

Also, physiological parameter measurement order information may beutilized to determine the priority. For example, when an image of afirst subject is acquired first and then an image of a second subject isacquired, a physiological parameter of the first subject may bemeasured, but the present invention is not limited thereto.

Also, the smart mirror device operating method 7600 according to anembodiment may include an operation of outputting the acquiredphysiological parameter (S7640).

In this case, the physiological parameter may be a physiologicalparameter of a person determined according to the priority.

Also, while the physiological parameter is being output, an indicatorfor a person whose physiological parameter is measured may be output.For example, a guide region for a person who is determined according tothe priority and whose physiological parameter is to be measured may beoutput, but the present invention is not limited thereto.

FIG. 61 is a diagram illustrating an operation of a smart mirror deviceaccording to an embodiment.

Referring to FIG. 61, a smart mirror device 7650 according to anembodiment may include an image sensor 7660 and a mirror display 7670.

In this case, the above description is applicable to the image sensor7660 and the mirror display 7670, and thus a redundant descriptionthereof will be omitted.

According to an embodiment, referring to FIG. 61, a first person 7680and a second person 7690 may be positioned in the measurement range ofthe image sensor 7660.

Also, the first person 7680 may be subject to the physiologicalparameter measurement of the smart mirror device 7650.

For example, as shown in FIG. 61, when the first person 7680 ispositioned closer to the image sensor 7660 than the second person 7690,the first person 7680 may be subject to the physiological parametermeasurement, but the present invention is not limited thereto.

Also, for example, when in an image acquired by the image sensor 7660,an area occupied by an image of the first person 7680 is larger than anarea occupied by an image of the second person 7690, the first person7680 may be subject to the physiological parameter measurement, but thepresent invention is not limited thereto.

Also, for example, when only the first person 7680 is actuallyrecognized as a person by the smart mirror device 7650, the first person7680 may be subject to the physiological parameter measurement, but thepresent invention is not limited thereto.

Also, for example, when in an image acquired by the image sensor 7660,an area occupied by an image of the first person 7680 is closer to thecenter than an area occupied by an image of the second person 7690, thefirst person 7680 may be subject to the physiological parametermeasurement, but the present invention is not limited thereto.

Also, for example, when the priority of the first person 7680 is storedto be higher than the priority of the second person 7690, the firstperson 7680 may be subject to the physiological parameter measurement,but the present invention is not limited thereto.

Also, for example, when the first person 7680 is stored as a subject,the first person 7680 may be subject to the physiological parametermeasurement, but the present invention is not limited thereto.

Also, when a physiological parameter of the first person 7680 ismeasured, the physiological parameter of the first person 7680 may beoutput. For example, as shown in FIG. 61, a heart rate, an oxygensaturation level, and a blood pressure of the first person 7680 may beoutput, but the present invention is not limited thereto.

Also, when a physiological parameter of the first person 7680 ismeasured, an indicator for the first person 7680 may be output. Forexample, as shown in FIG. 61, a guide region for the first person 7680may be output, but the present invention is not limited thereto.

9.1.7.3 Various Embodiments of Smart Mirror Device Configured to MeasurePhysiological Parameters of Plurality of Subjects

When a plurality of people are present in a physiological parametermeasurement region of a smart mirror device, a physiological parametermay be measured according to priority.

FIG. 62 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment.

Referring to FIG. 62, a smart mirror device operating method 7700according to an embodiment may include at least one of an operation ofacquiring an image including a plurality of subjects (S7710), anoperation of acquiring physiological parameters of the plurality ofsubjects (S7720), and an operation of outputting the physiologicalparameters of the plurality of subjects (S7730), but the presentinvention is not limited thereto.

In detail, the smart mirror device operating method 7700 according to anembodiment may include an operation of acquiring an image including aplurality of subjects (S7610). The above description about imageacquisition is applicable to this operation, and thus a redundantdescription thereof will be omitted.

Also, the smart mirror device operating method 7700 according to anembodiment may include an operation of acquiring physiologicalparameters of the plurality of subjects (S7720). The above-describedphysiological parameter acquisition methods are applicable to thisoperation, and thus a redundant description thereof will be omitted.

According to an embodiment, a region for each of the plurality ofsubjects may be set to acquire the physiological parameters of theplurality of subjects. For example, when a first subject and a secondsubject are positioned within a physiological parameter measurementregion, a region for the first subject and a region for the secondsubject may be set.

Also, a plurality of physiological parameters for each of a plurality ofsubjects may be acquired. For example, a heart rate, an oxygensaturation level, and a blood pressure of each of a plurality ofsubjects may be acquired, but the present invention is not limitedthereto.

Also, different types of physiological parameters may be acquired forthe plurality of subjects. For example, a heart rate of a first subjectmay be acquired, and an oxygen saturation level and a blood pressure ofa second subject may be acquired, but the present invention is notlimited thereto.

Also, a sub-region for each of the plurality of subjects may be set toacquire the plurality of physiological parameters of the plurality ofsubjects. For example, a region for a first subject, a first sub-regionfor measuring a heart rate of the first subject, a second sub-region formeasuring an oxygen saturation level of the first subject, and a thirdsub-region for measuring a blood pressure of the first subject may beset, and a region for a second subject, a fourth sub-region formeasuring a heart rate of the second subject, a fifth sub-region formeasuring an oxygen saturation level of the second subject, and a sixthsub-region for measuring a blood pressure of the second subject may beset, but the present invention is not limited thereto.

Also, the first to sixth sub-regions may differ from each other and atleast partially overlap each other.

Also, the smart mirror device operating method 7700 according to anembodiment may include an operation of outputting the physiologicalparameters of the plurality of subjects (S7730).

In this case, the physiological parameters may include at least onephysiological parameter of each of the plurality of subjects.

Also, while the physiological parameters are being output, indicatorsfor the plurality of subjects may be output. For example, whenphysiological parameters of the first subject and the second subject areacquired, guide regions for the first subject and the second subject maybe output, but the present invention is not limited thereto.

Also, while the physiological parameters are being output, indicatorsindicating for which subjects the physiological parameters are acquiredmay be output. For example, an indicator for identifying a first subjectmay be output when a physiological parameter of the first subject isoutput, and an indicator for identifying a second subject may be outputwhen a physiological parameter of the second subject is output.

FIG. 63 is a diagram illustrating an operation of a smart mirror deviceaccording to an embodiment.

Referring to FIG. 63, a smart mirror device 7750 according to anembodiment may include an image sensor 7760 and a mirror display 7770.

In this case, the above description is applicable to the image sensor7760 and the mirror display 7770, and thus a redundant descriptionthereof will be omitted.

According to an embodiment, referring to FIG. 63, a first subject 7780and a second subject 7790 may be positioned in the measurement range ofthe image sensor 7760.

Also, a region for each of a plurality of subjects may be set in animage frame acquired to acquire physiological parameters of the firstsubject 7780 and the second subject 7790.

Also, physiological parameters of the first subject 7780 and the secondsubject 7790 may be acquired, For example, heart rates of the firstsubject 7780 and the second subject 7790 may be acquired, but thepresent invention is not limited thereto.

Also, a plurality of physiological parameters for each of the firstsubject 7780 and the second subject 7790 may be acquired. For example,as shown in FIG. 63, a heart rate, an oxygen saturation level, and ablood pressure for the first subject 7780 may be acquired, and a heartrate, an oxygen saturation level, and a blood pressure for the secondsubject 7790 may be acquired, but the present invention is not limitedthereto.

Also, for example, although not shown in FIG. 63, a heart rate of thefirst subject 7780 may be acquired, and an oxygen saturation level and ablood pressure of the second subject 7790 may be acquired, but thepresent invention is not limited thereto.

9.1.7.4 Various Embodiments of Smart Mirror Device Configured to OutputDifferent Information Depending on Situation

When a smart mirror device is placed at the entrance of a building,e.g., on a shoe rack, information a user needs in a situation ofentering the building may be different from information a user needs ina situation of exiting the building.

Accordingly, the smart mirror device may output and provide differentinformation to a user depending on the situation.

Also, the smart mirror device may output and provide a differentphysiological parameter to a user depending on the situation.

FIG. 64 is a diagram illustrating a smart mirror device operating methodaccording to an embodiment.

Referring to FIG. 64, a smart mirror device operating method 7800according to an embodiment may include at least one of an operation ofdetermining an information provision situation (S7810), an operation ofoutputting first information corresponding to a first situation (S7820),and an operation of outputting second information corresponding to asecond situation (S7830), but the present invention is not limitedthereto.

In detail, the smart mirror device operating method 7800 according to anembodiment may include an operation of determining an informationprovision situation (S7810).

In this case, the information provision situation may refer to asituation of a user of a smart mirror device. For example, theinformation provision situation may include a situation in which theuser of the smart mirror device enters a building or a situation inwhich the user of the smart mirror device exits a building, but thepresent invention is not limited thereto.

Also, the information provision situation may simply refer to asituation determined to provide information. For example, theinformation provision situation may include a first situation or asecond situation, but the present invention is not limited thereto.

Also, the information provision situation may simply refer to asituation associated with whether to determine a physiologicalparameter. For example, the information provision situation may includea situation in which there is no need to measure a physiologicalparameter and a situation in which there is a need to measure aphysiological parameter, but the present invention is not limitedthereto.

Also, the information provision situation may refer to a situationassociated with time. For example, the information provision situationmay include a before-noon situation or an after-noon situation, but thepresent invention is not limited thereto.

Also, the information provision situation may refer to a situationcorresponding to an external environment. For example, the informationprovision situation may include a situation corresponding to an externalsituation such as an external epidemic situation, but the presentinvention is not limited thereto.

Also, the smart mirror device may include an additional sensor in orderto determine the information provision situation. For example, the smartmirror device may additionally include a motion detection sensor inorder to determine a user access situation, determine that the firstsituation is that a user's motion is detected first by a first motiondetection sensor, and determine that the second situation is that auser's motion is detected first by a second motion detection sensor, butthe present invention is not limited thereto.

Also, an image sensor included in the smart mirror device may be used todetermine the information provision situation. For example, an imageframe acquired through the image sensor may be used to determine theuser access situation. In detail, in a plurality of image framesacquired in time-series, the smart mirror device may determine that thefirst situation is that a region corresponding to the user becomes closeto a central part in a first aspect and may determine that the secondsituation is that the region corresponding to the user becomes close tothe central part in a second aspect, but the present invention is notlimited thereto.

Also, an image sensor included in the smart mirror device may be used todetermine the information provision situation. For example, an imageframe acquired through the image sensor may be used to determine whetherthere is a need for physiological parameter measurement and informationprovision. In detail, in a plurality of image frames acquired intime-series, the smart mirror device may determine that the firstsituation is that a movement speed of the region corresponding to theuser exceeds a reference value and may determine that the secondsituation is that the movement speed is less than or equal to thereference value, but the present invention is not limited thereto.

Also, time information may be used to determine the informationprovision situation. For example, the smart mirror device may determinethat the first situation is before a reference time and may determinethat the second situation is after the reference time, but the presentinvention is not limited thereto.

Also, recognition information may be used to determine the informationprovision situation. For example, face recognition information of peoplemay be used to determine whether physiological parameter measurement andinformation provision are necessary. In detail, according to a result offace recognition, the smart mirror device may determine that the firstsituation is that a user is registered and may determine that the secondsituation is that a user is not registered, but the present invention isnot limited thereto.

Also, movement direction information of a subject may be used todetermine the information provision situation. For example, movementdirection information of a subject detected to determine the informationprovision situation or detected using an external sensor may be used. Indetail, the smart mirror device may determine that the first situationis that the subject moves in a first direction and may determine thatthe second situation is that the subject moves in a second situation,but the present invention is not limited thereto.

Also, a smart mirror device operating method 7800 according to anembodiment may include an operation of outputting first informationcorresponding to the first situation (S7820) and an operation ofoutputting second information corresponding to the second situation(S7830), but the present invention is not limited thereto.

In this case, when the first situation includes a situation in which abuilding is exited and the second situation includes a situation inwhich a building is entered, the first information and the secondinformation may be different from each other.

For example, the first information may include external weatherinformation, today's schedule information, product informationcorresponding to external weather, and the like. However, the presentinvention is not limited thereto, and the first information may beinformation corresponding to the first situation.

Also, the second information may include internal temperatureinformation, internal humidity information, security information,internal air quality information, and the like. However, the presentinvention is not limited thereto, and the second information may beinformation corresponding to the second situation.

Also, physiological parameter information may be included in at leastone of the first information and the second information, but the presentinvention is not limited thereto.

Also, a physiological parameter included in the first information and aphysiological parameter included in the second information may bedifferent from each other. For example, a heart rate may be output whena physiological parameter of a first subject is provided, and a bloodpressure may be output when a physiological parameter of a secondsubject is provided, but the present invention is not limited thereto.

FIGS. 65 and 66 are diagrams illustrating an operation of a smart mirrordevice according to an embodiment.

Referring to FIGS. 65 and 66, a smart mirror device 7850 according to anembodiment may include an image sensor 7860 and a mirror display 7870.

In this case, the above description is applicable to the image sensor7860 and the mirror display 7870, and thus a redundant descriptionthereof will be omitted.

According to an embodiment, FIG. 65 may be a diagram illustrating theoperation of the smart mirror device 7850 in a first situation, and FIG.66 may be a diagram illustrating the operation of the smart mirrordevice 7850 in a second situation.

Also, information output in the first situation may be different frominformation output in the second situation. The above description isapplicable to this situation, and thus a redundant description thereofwill be omitted.

Also, an external sensor 7861 may be used to determine the firstsituation and the second situation. However, the present invention isnot limited thereto, and the above examples may be applied to determinethe first situation and the second situation.

Also, although not shown in FIGS. 65 and 66, the information output inthe first situation and the information output in the second situationmay at least partially overlap each other. For example, physiologicalparameters may be included in both of the information output in thefirst situation and the information output in the second situation, butthe present invention is not limited thereto.

9.2 Various Embodiments of Physiological Parameter Measurement DevicePlaced in Vehicle

The above-described physiological parameter acquisition method anddrowsiness detection method may be used in various applications. Forexample, the physiological parameter acquisition method or drowsinessdetection method may be used in a physiological parameter measurementdevice placed in a vehicle.

Also, the physiological parameter measurement device may perform anoperation corresponding to a physiological parameter or physiologicalinformation acquired according to the physiological parameteracquisition method.

Also, the physiological parameter measurement device may perform anoperation corresponding to drowsiness information acquired according tothe drowsiness detection method.

Also, the operation of the physiological parameter measurement devicewill be described below, but the operation to be described may beperformed even by other processors such as an ECU mounted on a vehicleor may be performed even by a processor included in a server.

Also, a device that will be described as the physiological parametermeasurement device may refer to an image acquisition device. In thiscase, the operation to be described may be performed by other processorssuch as an ECU or a processor included in a server.

FIG. 67 is a diagram illustrating a physiological parameter measurementdevice according to an embodiment.

Referring to FIG. 67, a physiological parameter measurement device 8010according to an embodiment may be placed in a vehicle 8000 andconfigured to measure a physiological parameter of an occupant 8020 inthe vehicle 8000.

In this case, the physiological parameter measurement device 8010 may beplaced at various positions of the vehicle 8000.

For example, the physiological parameter measurement device 8010 may beplaced on an overhead console, a sun visor, a rearview mirror, adashboard, an instrument panel, a steering wheel, a center fascia, aconsole box, etc. of the vehicle 8000, but the present invention is notlimited thereto.

Also, the occupant 8020 may be a driver. However, the present inventionis not limited thereto, and the occupant 8020 may be a person in avehicle such as a passenger.

Also, when the occupant 8020 includes a plurality of people, thephysiological parameter measurement device 8010 may measurephysiological parameters of the plurality of people.

Also, the physiological parameter measurement device 8010 may include animage sensor and acquire a physiological parameter on the basis of animage frame acquired from the image sensor. However, the presentinvention is not limited thereto, and a contact-type sensor or the likemay be used.

Also, when the physiological parameter measurement device 8010 includesan image sensor, the image sensor may be provided as a visible camerafor acquiring a visible light image, an IR camera for acquiring aninfrared image, and the like. However, the present invention is notlimited thereto, and a hybrid-type camera for acquiring a visible lightimage and an infrared image may be provided.

Also, the physiological parameter measurement device 8010 may operate ina mode changed according to the amount of external light. For example,the physiological parameter measurement device 8010 may operate in afirst mode during the daytime when there is a great deal of externallight and may operate in a second mode during the night when there islittle external light, but the present invention is not limited thereto.

Also, for example, the physiological parameter measurement device 8010may operate in a first mode in which a physiological parameter isacquired based on an RGB image when the external light is higher than orequal to a reference value and may operate in a second mode in which aphysiological parameter is acquired based on an IR image when theexternal light is lower than or equal to a reference value, but thepresent invention is not limited thereto.

FIG. 68 is a diagram illustrating a physiological parameter measurementdevice according to an embodiment.

Referring to FIG. 68, a physiological parameter measurement deviceaccording to an embodiment may be placed at various positions of avehicle.

For example, as shown in FIG. 68, a first physiological parametermeasurement device 8011 may be placed on a driver seat dashboard, asecond physiological parameter measurement device 8012 may be placed ona passenger seat dashboard, and a third physiological parametermeasurement device 8013 may be placed on a rearview mirror, but thepresent invention is not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may be placed inside thevehicle.

Also, at least one of the -first to third physiological parametermeasurement devices 8011, 8012, and 8013 may acquire a physiologicalparameter of the driver.

For example, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may measure a physiologicalparameter, such as a heart rate, a blood pressure, an oxygen saturationlevel, and a core temperature, of the driver, but the present inventionis not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may perform an operationcorresponding to the acquired physiological parameter of the driver.

For example, when the acquired physiological parameter of the driver isabnormal, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may perform an operation ofproviding information about nearby hospitals, but the present inventionis not limited thereto.

Also, for example, when the acquired physiological parameter of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

Also, for example, when the acquired physiological parameter of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may generate ahardware alarm such as sound and vibration, but the present invention isnot limited thereto.

Also, for example, when the acquired physiological parameter of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 performs an operationof changing a vehicle to an autonomous driving mode, but the presentinvention is not limited thereto.

Also, at least one of the first, second, and third physiologicalparameter measurement devices 8011, 8012, and 8013 may acquirephysiological information of the driver.

For example, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may acquire the physiologicalinformation of the driver, such as drowsiness information and conditioninformation of the driver, but the present invention is not limitedthereto.

Also, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may perform an operationcorresponding to the acquired physiological information of the driver.

For example, when the acquired drowsiness information of the driver isabnormal, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may generate a hardware alarmsuch as sound and vibration, but the present invention is not limitedthereto.

Also, for example, when the acquired drowsiness information of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation of providing information about nearby rest areas, but thepresent invention is not limited thereto.

Also, for example, when the acquired drowsiness information of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

Also, for example, when the acquired drowsiness information of thedriver is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 performs an operationof changing a vehicle to an autonomous driving mode, but the presentinvention is not limited thereto.

Also, for example, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation of selecting an appropriate song according to the acquiredcondition information of the driver, but the present invention is notlimited thereto.

Also, for example, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation of providing information about appropriate content accordingto the acquired condition information of the driver, but the presentinvention is not limited thereto.

Also, at least one of the first, second, and third physiologicalparameter measurement devices 8011, 8012, and 8013 may acquire aphysiological parameter of a passenger.

For example, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may measure a physiologicalparameter, such as a heart rate, a blood pressure, an oxygen saturationlevel, and a core temperature, of the passenger, but the presentinvention is not limited thereto.

Also, at least one of the first, second, and third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation corresponding to the acquired physiological parameter of thepassenger.

For example, when the acquired physiological parameter of the passengeris abnormal, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may perform an operation ofproviding an alarm to the driver, but the present invention is notlimited thereto.

Also, for example, when the acquired physiological parameter of thepassenger is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation of providing information about nearby hospitals, but thepresent invention is not limited thereto.

Also, for example, when the acquired physiological parameter of thepassenger is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

Also, for example, when the acquired physiological parameter of thepassenger is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 performs an operationof changing a vehicle to an autonomous driving mode, but the presentinvention is not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may acquire physiologicalinformation of the passenger.

For example, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may acquire the physiologicalinformation of the passenger, such as drowsiness information andcondition information of the driver, but the present invention is notlimited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8011, 8012, and 8013 may perform an operationcorresponding to the acquired physiological information of thepassenger.

For example, when the acquired physiological information of thepassenger is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may perform anoperation of providing relevant information to the driver, but thepresent invention is not limited thereto.

Also, for example, when the acquired physiological information of thepassenger is abnormal, at least one of the first to third physiologicalparameter measurement devices 8011, 8012, and 8013 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

FIG. 69 is a diagram illustrating a physiological parameter measurementdevice placed on an autonomous vehicle according to an embodiment.

Referring to FIG. 69, a physiological parameter measurement deviceaccording to an embodiment may be placed at various positions of avehicle.

For example, as shown in FIG. 69, a first physiological parametermeasurement device 8014 may be placed on one side of a table, a secondphysiological parameter measurement device 8015 may be placed on theother side, and a third physiological parameter measurement device 8016may be placed on the ceil of an autonomous vehicle, but the presentinvention is not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may be placed inside thevehicle.

Also, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may acquire a physiologicalparameter of an occupant.

For example, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may measure a physiologicalparameter, such as a heart rate, a blood pressure, an oxygen saturationlevel, and a core temperature, of the occupant, but the presentinvention is not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may perform an operationcorresponding to the acquired physiological parameter of the occupant.

For example, when the acquired physiological parameter of the occupantis abnormal, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may perform an operation ofproviding information about nearby hospitals, but the present inventionis not limited thereto.

Also, for example, when the acquired physiological parameter of theoccupant is abnormal, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may perform anoperation of driving to a nearby hospital, but the present invention isnot limited thereto.

Also, for example, when the acquired physiological parameter of theoccupant is abnormal, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

Also, for example, when the acquired physiological parameter of theoccupant is abnormal, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may generate ahardware alarm such as sound and vibration, but the present invention isnot limited thereto.

Also, for example, when the acquired physiological parameter of theoccupant is abnormal, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may display relevantinformation, but the present invention is not limited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may perform an operationcorresponding to a result of recognizing the occupant.

For example, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may recognize the face of theoccupant and display information of the occupant when the face isrecognized, but the present invention is not limited thereto.

Also, for example, only when the face of the occupant is recognized andthe physiological parameter is acquired, may at least one of the firstto third physiological parameter measurement devices 8014, 8015, and8016 display information of the occupant, but the present invention isnot limited thereto.

Also, for example, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may acquirephysiological information of the occupant.

For example, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may acquire the physiologicalinformation of the occupant, such as drowsiness information andcondition information of the occupant, but the present invention is notlimited thereto.

Also, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may perform an operationcorresponding to the acquired physiological information of the occupant.

For example, when the acquired physiological information of the occupantis abnormal, at least one of the first to third physiological parametermeasurement devices 8014, 8015, and 8016 may generate a hardware alarmsuch as sound and vibration, but the present invention is not limitedthereto.

Also, for example, when the acquired physiological information of theoccupant is abnormal, at least one of the first to third physiologicalparameter measurement devices 8014, 8015, and 8016 may transmit relevantinformation to another mobile device, but the present invention is notlimited thereto.

Also, although not shown FIGS. 68 and 69, the physiological parametermeasurement device may be placed in an ambulance or the like and may beused to measure a patient's physiological parameter and acquirephysiological information.

9.2.1 Various Embodiments of Physiological Parameter Measurement DevicePerforming Operation Corresponding to Drowsiness Information

FIG. 70 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 70, a physiological parameter measurement deviceoperating method 8100 according to an embodiment may include anoperation of acquiring a physiological parameter (S8110) and anoperation of determining a drowsiness level (S8120). An alarm may beoutput according to the determined drowsiness level. For example, asshown in FIG. 70, a first alarm S8130, a second alarm S8140, and a thirdalarm S8150 may be output, but the present invention is not limitedthereto.

In this case, the above-described physiological parameter acquisitionmethod or the like is applicable to the operation of acquiring aphysiological parameter (S8110), and thus a redundant descriptionthereof will be omitted.

Also, the above-described drowsiness detection method or the like isapplicable to the operation of determining the drowsiness level (S8120),and thus a redundant description thereof will be omitted.

Also, the first alarm S8130 may be an alarm that is output at the lowestdrowsiness level. For example, the first alarm S8130 may include avisual alarm, but the present invention is not limited thereto.

Also, the second alarm S8140 may be an alarm that s output at a middlelevel.

For example, the second alarm S8140 may include an auditory alarm thathas a weak intensity or a long cycle, but the present invention is notlimited thereto.

Also, the third alarm S8150 may be an alarm that is output at thehighest drowsiness level.

For example, the third alarm S8150 may include an auditory alarm thathas a strong intensity or a short cycle, but the present invention isnot limited thereto.

However, the above-described first to third alarms S8130, S8140, andS8150 are not limited to the above-described examples, and it is obviousthat examples of outputting a different alarm depending on thedrowsiness level are applicable.

Also, the physiological parameter measurement device operating method8100 according to an embodiment may include an operation of transmittingdrowsiness information (S8160).

Also, the operation of transmitting the drowsiness information (S8160)may be performed at only at least some drowsiness levels. For example,the operation of transmitting the drowsiness information (S8160) may notbe performed at a first drowsiness level and may be performed at secondand third drowsiness levels, but the present invention is not limitedthereto.

Also, the operation of transmitting the drowsiness information (S8160)may include an operation of transmitting the drowsiness information toan administrator. For example, information regarding a drowsiness levelof a driver of a vehicle may be transmitted to an administrator whoperforms vehicle dispatch and may be reflected in the vehicle dispatch.

Also, the operation of transmitting the drowsiness information (S8160)may include an operation of transmitting the drowsiness information to adriver's mobile terminal. For example, drowsiness information andinformation on a corresponding operating section may be transmitted to adriver's mobile terminal so that the driver can pay attention to thenext operating.

Also, the operation of transmitting the drowsiness information (S8160)may include an operation of transmitting the drowsiness information to aroad administrator. For example, information on a driver's drowsinesslevel and information on an operating section may be transmitted andused to produce a drowsiness map or the like.

9.2.2 Various Embodiments of Driving Scheduling Method UsingPhysiological Parameter and Physiological Information

FIG. 71 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 71, a physiological parameter measurement deviceoperating method 8200 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter (S8210), anoperation of acquiring physiological information (S8220), andcalculating a driving scheduling parameter (S8230), but the presentinvention is not limited thereto.

In this case, the above-described physiological parameter acquisitionmethod is applicable to the operation of acquiring a physiologicalparameter (S8210), and thus a redundant description thereof will beomitted.

Also, the above-described physiological information acquisition methodis applicable to the operation of acquiring the physiologicalinformation (S8220), and thus a redundant description thereof will beomitted.

Also, the driving scheduling parameter may be a parameter for assistingin driving scheduling.

In this case, the driving scheduling parameter may be acquired based onthe physiological parameter and the physiological information. Forexample, the driving scheduling parameter may be acquired based on heartrate information, drowsiness information, and the like of a driver, butthe present invention is not limited thereto.

Also, the driving scheduling parameter may be acquired based on vehicleoperating information. For example, the driving scheduling parameter maybe acquired based on an operating distance, an operating time, thenumber of times of operating, and the like of a vehicle.

Also, the driving scheduling parameter may be acquired based on aphysiological parameter and physiological information of a driver andvehicle operating information. For example, the driving schedulingparameter may be acquired based on drowsiness information of a driver,an operating distance and the number of times of operating of a vehicle,and the like, but the present invention is not limited thereto.

FIG. 72 is a diagram illustrating a driving scheduling assistance deviceoperating method using a physiological parameter measurement deviceaccording to an embodiment.

Referring to FIG. 72, a driving scheduling assistance device may acquireinformation from a first physiological parameter measurement deviceplaced in a first vehicle and a second physiological parametermeasurement device placed in a second vehicle.

In detail, the first physiological parameter measurement device mayacquire a physiological parameter and physiological information of afirst driver in the first vehicle. For example, the first physiologicalparameter measurement device may acquire a heart rate and drowsinessinformation of the first driver, but the present invention is notlimited thereto.

Also, the second physiological parameter measurement device may acquirea physiological parameter and physiological information of a seconddriver in the second vehicle. For example, the second physiologicalparameter measurement device may acquire a heart rate and drowsinessinformation of the second driver, but the present invention is notlimited thereto.

Also, as shown in FIG. 72, the driving scheduling assistance device mayacquire the physiological parameter and physiological information of thefirst driver from the first physiological parameter measurement deviceand acquire the physiological parameter and physiological information ofthe second driver from the second physiological parameter measurementdevice, but the present invention is not limited thereto.

Also, as shown in FIG. 72, the driving scheduling assistance device mayacquire first operating information for first operating of a firstvehicle and second operating information for second operating of asecond vehicle.

In this case, the first operating information and the second operatinginformation may include an operating distance, the number of times ofoperating, an operating time, and the like, but the present invention isnot limited thereto.

Also, the driving scheduling assistance device may calculate a drivingscheduling parameter on the basis of the physiological parameter andphysiological information of the first driver, the physiologicalparameter and physiological information of the second driver, the firstoperating information of the first vehicle, and the second operatinginformation of the second vehicle.

For example, the driving scheduling assistance device may calculate afirst driving scheduling parameter on the basis of first drowsinessinformation of the first driver and the first operating information ofthe first vehicle and calculate a second driving scheduling parameter onthe basis of second drowsiness information of the second driver and thesecond operating information of the second vehicle, but the presentinvention is not limited thereto.

Also, for example, when a first operating distance included in the firstoperating information is equal to a second operating distance includedin the second operating information, a driving scheduling parameter maybe calculated so that one of the first driver and the second driver, whohas a lower drowsiness parameter corresponding to drowsinessinformation, can perform third operating.

Also, for example, the first driving scheduling parameter may becalculated by adding different weights to the first drowsinessinformation and the first operating information, which are forcalculating the first driving scheduling parameter.

In detail, the first driving scheduling parameter may be calculated byadding the larger weight to the first operating information, but thepresent invention is not limited thereto.

Also, for example, the second driving scheduling parameter may becalculated by adding different weights to the second drowsinessinformation and the second operating information, which are forcalculating the second driving scheduling parameter.

In detail, the second driving scheduling parameter may be calculated byadding the larger weight to the second operating information, but thepresent invention is not limited thereto.

Also, the driving scheduling assistance device may use the calculateddriving scheduling parameter to perform driving scheduling for the thirdoperating, but the present invention is not limited thereto.

Also, the driving scheduling assistance device may display thecalculated driving scheduling parameter, but the present invention isnot limited thereto.

9.2.3 Various Embodiments of Physiological Parameter Measurement DeviceUsed to Unlock Vehicle

FIG. 73 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 73, a physiological parameter measurement deviceoperating method 8300 according to an embodiment may include at leastsome of an operation of recognizing a subject's face (S8310), anoperation of acquiring a physiological parameter of the subject (S8320),and an operation of unlocking at least some operations of a vehicle(S8330), but the present invention is not limited thereto.

In this case, the above description arid a typical face recognitionmethod are applicable to the operation of recognizing the subject's face(S8310), and thus a redundant description thereof will be omitted.

Also, the above-described physiological parameter acquisition method isapplicable to the operation of acquiring a physiological parameter ofthe subject (S8320), arid thus a redundant description thereof will beomitted.

Also, when the subject's face is recognized and the physiologicalparameter is acquired, the operation of unlocking at least someoperations of the vehicle (S8330) may be performed.

For example, when the subject's face is recognized but the physiologicalparameter is not acquired, the operation of unlocking at least someoperations of the vehicle (S8330) may not be performed, but the presentinvention is not limited thereto.

Also, in this case, it is possible to prevent unlocking caused by stolenphotos or the like and strengthen vehicle security.

Also, when the recognized face of the subject matches an ID stored in aserver and the physiological parameter is acquired, the operation ofunlocking at least some operations of the vehicle (S8330) may beperformed.

For example, when the subject's face is recognized and the physiologicalparameter is acquired but the recognized face of the subject does notmatch an ID stored in the server, the operation of unlocking at leastsome operations of the vehicle (S8330) may not be performed, but thepresent invention is not limited thereto.

Also, when the subject's face is recognized and the physiologicalparameter matches an ID stored in a server, the operation of unlockingat least some operations of the vehicle (S8330) may be performed.

For example, when the subject's face is recognized and the physiologicalparameter is acquired but the acquired physiological parameter of thesubject does not match an ID stored in the server, the operation ofunlocking at least some operations of the vehicle (S8330) may not beperformed, but the present invention is not limited thereto.

Also, the locking of at least some operations of the vehicle may be thelocking on the start-up of the vehicle, but the present invention is notlimited thereto.

9.2.4 Various Embodiments of Driving Parameter Calculation DeviceOperating Method Using Physiological Parameter Measurement Device

FIG. 74 is a flowchart illustrating a driving parameter calculationdevice operating method according to an embodiment.

Referring to FIG. 74, a driving parameter calculation device operatingmethod 8350 according to an embodiment may include at least some of anoperation of acquiring operating information (S8360), an operation ofacquiring drowsiness information of a subject (S8370), and an operationof calculating a driving parameter of the subject (S8380), but thepresent invention is not limited thereto.

In this case, the driving parameter may be a comprehensive evaluationparameter for a driver's operating or a numerical value calculated incomprehensive consideration of various pieces of information, but thepresent invention is not limited thereto.

Also, in this case, the operating information may include operatingdistance information, information on the number of times of operating,operating time information, operating section information, roadinformation, and the like for an operating vehicle, but the presentinvention is not limited thereto.

Also, the above-described drowsiness detection method is applicable tothe operation of acquiring the drowsiness information of the subject(S8370), and thus a redundant description thereof will be omitted.

Also, the operation of calculating the driving parameter of the subject(S8380) may include an operation of calculating a driving parameter onthe basis of the operating information and the drowsiness information.

For example, the driving parameter may be calculated in comprehensiveconsideration of the number of times drowsiness occurs during a drivingtime, the degree of drowsiness, a method of coping with drowsiness, andthe like, but the present invention is not limited thereto.

Also, a driver may be rewarded on the basis of the calculated drivingparameter. For example, a driver may be rewarded on the basis of adriving parameter calculated based on driving time information ofdrivers who have been driving for a long time and drowsiness informationindicating that drowsiness does not occur, but the present invention isnot limited thereto.

Also, a driving parameter of a subject may be calculated in various wayson the basis of an operating time and drowsiness information in additionto the above examples.

9.3 Various Embodiments of Physiological Parameter Measurement DevicePlaced in Infant Monitoring Device

Monitoring infants using image sensors such as a camera is becomingessential for their health and safety.

In addition, monitoring the health and safety of infants by measuringtheir physiological parameters is also becoming essential.

However, when a separate sensor is attached or a separate contact-typesensor is placed to monitor physiological parameters of infants, thismay cause inconvenience to the infants, thereby limiting the role of acomfortable and safe shelter.

Accordingly, by monitoring infants using image sensors such as a cameraand simultaneously monitoring their physiological parameters or the likeusing image sensors such as a camera, it is possible to protect thehealth and safety of the infants and provide a comfortable and safeshelter for the infants.

FIG. 75 is a diagram illustrating an infant monitoring device accordingto an embodiment.

Referring to FIG. 75, an infant monitoring device 8400 according to anembodiment may acquire a physiological parameter of an infant 8410 in acontactless manner.

In detail, the infant monitoring device 8400 may include an image sensorand acquire a physiological parameter on the basis of an image frameacquired from the image sensor, but the present invention is not limitedthereto.

Also, the above description is applicable to the infant monitoringdevice 8400 acquiring a physiological parameter on the basis of an imageframe, and thus a redundant description thereof will be omitted.

Also, the infant monitoring device 8400 may store or transmit an imageacquired through the image sensor.

For example, the infant monitoring device 8400 may acquire an image thatat least partially includes an infant 8410 to be monitored and may storethe acquired image or transmit the acquired image to a user's mobilephone.

Also, the infant monitoring device 8400 may detect motion of the infant8410. For example, the infant monitoring device 8400 may detect whetherthe infant 8410 turns over, but the present invention is not limitedthereto.

Also, the infant monitoring device 8400 may detect sound made by theinfant 8410. For example, the infant monitoring device 8400 may detectthe crying sound of the infant 8410, but the present invention is notlimited thereto.

Also, the infant monitoring device 8400 may generate sound. For example,when an input for sound is received from an input device, the infantmonitoring device 8400 may output sound corresponding to the input. Indetail, when the voice of the parents of the infant 8410 is input, theinfant monitoring device 8400 may output the voice, but the presentinvention is not limited thereto.

Also, the infant monitoring device 8400 may provide illumination. Forexample, when an input for illumination is received from an inputdevice, the infant monitoring device 8400 may provide illuminationcorresponding to the input, but the present invention is not limitedthereto.

Also, the infant monitoring device 8400 may control the movement of acradle with the infant 8410. For example, when an input for a cradle isreceived from an input device, the infant monitoring device 8400 mayallow the movement of the cradle corresponding to the input, but thepresent invention is not limited thereto.

Also, the infant monitoring device 8400 may output an alarm. Forexample, when an event occurs in relation to the infant 8410, the infantmonitoring device 8400 may output an auditory alarm. However, thepresent invention is not limited thereto, and the infant monitoringdevice 8400 may output visual and tactile alarms and may transmitrelevant information to a user's mobile terminal.

Also, the infant monitoring device 8400 may perform various operationsfor monitoring the infant 8410 in addition to the above-describedexamples.

9.3.1 Various Embodiments of Infant Monitoring Device PerformingOperation Corresponding to Event Occurrence

FIG. 76 is a diagram illustrating the occurrence of an event in relationto an infant.

Referring to FIG. 76, it can be seen that various events may occur inrelation to an infant.

Referring to FIG. 76A, it can be seen that an event in which an infantturns over may occur.

Also, when an infant turns over, his or her breathing passage may beblocked, which can pose a great risk to the infant.

Also, referring to FIG. 76B, it can be seen that an event in which aninfant moves out of a monitoring region may occur.

In this case, an infant may have been moved by a nurse at a postpartumcare center or the like but may also have been moved by an unintendedvisitor.

Also, although not shown in FIG. 76, various events which can threatenthe health and safety of an infant may occur.

Accordingly, there may be a need for an infant monitoring device capableof monitoring, tracking, and managing events.

FIG. 77 is a flowchart illustrating an infant monitoring deviceoperating method according to an embodiment.

Referring to FIG. 77, an infant monitoring device operating method 8500according to an embodiment may include at least some of an operation ofacquiring a physiological parameter (S8510), an operation of determiningwhether an event has occurred (S8520), and an operation of performing anoperation corresponding to the event (S8530), but the present inventionis not limited thereto.

In this case, the above-described physiological parameter acquisitionmethod is applicable to the operation of acquiring a physiologicalparameter (S8510), and thus a redundant description thereof will beomitted.

According to an embodiment, the operation of determining whether anevent has occurred (S8520) may be performed based on the acquiredphysiological parameter.

For example, when no physiological parameter is acquired through theoperation of acquiring a physiological parameter (S8510), it may bedetermined that an event has occurred. In detail, for example, when aninfant whose physiological parameter is to be acquired turns over, nophysiological parameter may be acquired, and in this case, it may bedetermined that an event has occurred, but the present invention is notlimited thereto.

Also, for example, when the physiological parameter acquired through theoperation of acquiring a physiological parameter (S8510) is abnormal, itmay be determined that an event has occurred. In detail, for example,when a heart rate of an infant whose physiological parameter is to beacquired is abnormal, it may be determined that an event has occurred,but the present invention is not limited thereto.

Also, according to an embodiment, the operation of determining whetheran event has occurred (S8520) may be performed based on a plurality ofphysiological parameters.

For example, it may be determined that a first event has occurred whenat least one physiological parameter is not acquired through theoperation of acquiring a physiological parameter (S8510), and it may bedetermined that a second event has occurred when all physiologicalparameters are not acquired, but the present invention is not limitedthereto.

Also, for example, it may be determined that a first event has occurredwhen at least one physiological parameter acquired through the operationof acquiring a physiological parameter (S8510) is abnormal, and it maybe determined that a second event has occurred when all physiologicalparameters are abnormal, but the present invention is not limitedthereto.

Also, according to an embodiment, the operation of determining whetheran event has occurred (S8520) may be performed based on a change in thephysiological parameter.

For example, it may be determined that an event has occurred when asudden change in the physiological parameter acquired through theoperation of acquiring a physiological parameter (S8510) is detected. Indetail, for example, when an infant whose physiological parameter is tobe acquired changes from a sleep state to an awake state, the heart rateof the infant may increase, and in this case, it may be determined thatan event has occurred, but the present invention is not limited thereto.

Also, according to an embodiment, the operation of performing theoperation corresponding to the event (S8530) may include an operation ofperforming an operation corresponding to determined event information.

In this case, the operation corresponding to the event may include ahardware alarm such as visual, auditory, and tactile alarms. Forexample, when it is determined that an event has occurred, the infantmonitoring device may output an auditory sound alarm, but the presentinvention is not limited thereto.

Also, the operation corresponding to the event may include an operationof recording an image. For example, when it is determined that an eventhas occurred, the infant monitoring device may operate to record animage, but the present invention is not limited thereto.

Also, the operation corresponding to the event may include an operationof transmitting information on the event. For example, when it isdetermined that an event has occurred, the infant monitoring device maytransmit the information on the event to a user's mobile terminal, butthe present invention is not limited thereto.

Also, the operation corresponding to the event may include an operationof storing information on an event occurrence time point. For example,when it is determined that an event has occurred, the infant monitoringdevice may store information on an event occurrence time point, but thepresent invention is not limited thereto.

Also, the operation corresponding to the event may include variousoperations corresponding to the event that has occurred other than theabove examples.

FIG. 78 is a flowchart illustrating an infant monitoring deviceoperating method according to an embodiment.

Referring to FIG. 78, an infant monitoring device operating method 8550according to an embodiment may include at least some of an operation ofacquiring a physiological parameter (S8560), an operation of determiningwhether an event has occurred (S8570), an operation of performing afirst operation corresponding to a first event (S8580), and an operationof performing a second operation corresponding to a second event(S8590), but the present invention is not limited thereto.

In this case, the above-described physiological parameter acquisitionmethod is applicable to the operation of acquiring a physiologicalparameter (S8560), and thus a redundant description thereof will beomitted.

According to an embodiment, the operation of determining whether anevent has occurred (S8570) may be performed based on the acquiredphysiological parameter.

For example, when no physiological parameter is acquired through theoperation of acquiring a physiological parameter (S8560), it may bedetermined that an event has occurred. In detail, for example, when aninfant whose physiological parameter is to be acquired turns over, nophysiological parameter may be acquired, and in this case, it may bedetermined that an event has occurred, but is not limited thereto.

Also, for example, when the physiological parameter acquired through theoperation of acquiring the physiological parameter (S8560) is abnormal,it may be determined that an event has occurred. In detail, for example,when a heart rate of an infant whose physiological parameter is to beacquired is abnormal, it may be determined that an event has occurred,but the present invention is not limited thereto.

Also, according to an embodiment, the operation of determining whetheran event has occurred (S8570) may be performed based on a plurality ofphysiological parameters.

For example, it may be determined that a first event has occurred whenat least one physiological parameter is not acquired through theoperation of acquiring the physiological parameter (S8560), and it maybe determined that a second event has occurred when all physiologicalparameters are not acquired, but the present invention is not limitedthereto.

Also, for example, it may be determined that a first event has occurredwhen at least one physiological parameter acquired through the operationof acquiring the physiological parameter (S8560) is abnormal, and it maybe determined that a second event has occurred when all physiologicalparameters are abnormal, but the present invention is not limitedthereto.

Also, according to an embodiment, the operation of determining whetheran event has occurred (S8570) may be performed based on a change in thephysiological parameter.

For example, it may be determined that an event has occurred when asudden change in the physiological parameter acquired through theoperation of acquiring the physiological parameter (S8560) is detected.In detail, for example, when an infant whose physiological parameter isto be acquired changes from a sleep state to an awake state, the heartrate of the infant may increase, and in this case, it may be determinedthat an event has occurred, but the present invention is not limitedthereto.

Also, according to an embodiment, the operation of performing the firstoperation corresponding to the first event (S8580) and the operation ofperforming the second operation corresponding to the second event(S8590) may include an operation of performing an operationcorresponding to determined event information.

In this case, the first operation and the second operation may include ahardware alarm such as visual, auditory, and tactile alarms. Forexample, when it is determined that an event has occurred, the infantmonitoring device may output an auditory sound alarm, but the presentinvention is not limited thereto.

Also, the first operation and the second operation may include anoperation of recording an image. For example, when it is determined thatan event has occurred, the infant monitoring device may operate torecord an image, but the present invention is not limited thereto.

Also, the first operation and the second operation may include anoperation of transmitting information on the event. For example, when itis determined that an event has occurred, the infant monitoring devicemay transmit the information on the event to a user's mobile terminal,but the present invention is not limited thereto.

Also, the first operation and the second operation may include anoperation of storing information on an event occurrence time. Forexample, when it is determined that an event has occurred, the infantmonitoring device may store information on an event occurrence timepoint, but the present invention is not limited thereto.

Also, the first operation and the second operation may be different fromeach other. However, these operations may be entirely or at leastpartially the same.

Also, the first operation and the second operation may include variousoperations corresponding to the first event and the second event otherthan the above-described examples.

9.3.2 Various Embodiments of Infant Monitoring System

FIG. 79 is a diagram showing a mobile application for implementing aninfant monitoring system according to an embodiment.

Referring to FIG. 79, a mobile application 8600 according to anembodiment may include at least some of a video display region 8610, aphysiological parameter display region 8620, an input button displayregion 8630, a video time display region 8640, and a display region 8650for a video recorded during an event time, but the present invention isnot limited thereto.

In this case, a video displayed in the video display region 8610 may bea video or image acquired from the image monitoring device, but thepresent invention is not limited thereto.

Also, a physiological parameter displayed in the physiological parameterdisplay region 8620 includes at least one physiological parameter. Forexample, as shown in FIG. 79, the physiological parameter may include aheart rate, an oxygen saturation level, a blood pressure, and the like,but the present invention is not limited thereto.

Also, a physiological parameter displayed in the physiological parameterdisplay region 8620 may be a physiological parameter acquired from aninfant monitoring device, but the present invention is not limitedthereto.

Also, a physiological parameter displayed in the physiological parameterdisplay region 8620 may be a physiological parameter acquired from atleast one sensor, but the present invention is not limited thereto.

Also, a physiological parameter displayed in the physiological parameterdisplay region 8620 may include physiological parameters acquired fromdifferent sensors. For example, a heart rate displayed in thephysiological parameter display region 8620 may be a physiologicalparameter acquired from an infant monitoring device, and a bloodpressure may be a physiological parameter acquired from an externalblood pressure sensor, but the present invention is not limited thereto.

Also, an input button displayed in the input button display region 8630may include at least one input button.

Also, the input button may include at least some of a video recordbutton, an illumination button, a talk button, a cradle shaking button,and an alarm button, but the present invention is not limited thereto.

Also, a video time displayed in the video time display region 8640 maybe time information of a recorded video.

Also, an event occurrence time point may be included in the video timedisplayed in the video time display region 8640. For example, as shownin FIG. 79, a first-event occurrence time point and a second-eventoccurrence time point may be included in the video time, but the presentinvention is not limited thereto.

Also, a thumbnail of the recorded video may be displayed in the displayregion 8650 for a video recorded during an event time.

Also, information on the recorded video may be displayed in the displayregion 8650 for a video recorded during an event time. For example, asshown in FIG. 79, time information of the recorded video or the like maybe displayed, but the present invention is not limited thereto.

However, the above-described examples and drawings are for illustrativepurposes only, and the present invention is not limited thereto. Variousforms of applications may be provided as an application for implementingthe infant monitoring system.

9.4 Various Embodiments of Physiological Parameter Measurement DevicePlaced in Reading Room

By measuring a physiological parameter and physiological information inspaces for personal work and study, such as a reading room, it ispossible to monitor individuals' concentration while continuouslymonitoring their health, thereby enabling efficient work or study.

Also, the concentration may be acquired based on a heart rate. Forexample, concentration information may be acquired based on a change,magnitude, and a change pattern of a heart rate, but the presentinvention is not limited thereto.

FIG. 80 is a diagram illustrating a physiological parameter measurementdevice placed in a reading room according to an embodiment.

Referring to FIG. 80, a physiological parameter measurement device 8700according to an embodiment may acquire a physiological parameter andphysiological information of a subject 8710.

In this case, the physiological parameter may include a heart rate, anoxygen saturation level, a blood pressure, a core temperature, or thelike, but the present invention is not limited thereto.

Also, the physiological information may include physiologicalinformation such as drowsiness information, condition information, andconcentration information, but the present invention is not limitedthereto.

Also, the above description is applicable to a method of acquiring thephysiological parameter and the physiological information, and thus aredundant description thereof will be omitted.

FIG. 81 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 81, a physiological parameter measurement deviceoperating method 8720 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter (S8730), anoperation of acquiring physiological information (S8740), and performingan operation corresponding to the physiological information (S8750).

In this case, the above description is applicable to the operation ofacquiring the physiological parameter (S8730) and the operation ofacquiring the physiological information (S8740), and thus a redundantdescription thereof will be omitted.

Also, the operation of performing the operation corresponding to thephysiological information (S8750) may include an operation of performingvarious operations corresponding to various pieces of physiologicalinformation.

For example, when the physiological information is drowsinessinformation, the physiological parameter measurement device may performan operation of outputting an alarm according to a drowsiness level, butthe present invention is not limited thereto.

Also, for example, when the physiological information is drowsinessinformation, the physiological parameter measurement device may performan operation of transmitting information to an administrator accordingto a drowsiness level, but the present invention is not limited thereto.

Also, for example, when the physiological information is drowsinessinformation, the physiological parameter measurement device may performan operation of transmitting information to a user's terminal accordingto a drowsiness level, but the present invention is not limited thereto.

Also, for example, when the physiological information is drowsinessinformation, the physiological parameter measurement device may performan operation of transmitting information to a scheduling deviceaccording to a drowsiness level, but the present invention is notlimited thereto.

Also, for example, when the physiological information is drowsinessinformation, the physiological parameter measurement device may performan operation of outputting an advisory text for taking a rest accordingto a drowsiness level, but the present invention is not limited thereto.

Also, for example, when the physiological information is conditioninformation, the physiological parameter measurement device may performan operation of outputting an alarm according to a condition, but thepresent invention is not limited thereto.

Also, for example, when the physiological information is conditioninformation, the physiological parameter measurement device may performan operation of transmitting information to an administrator accordingto a condition, but the present invention is not limited thereto.

Also, for example, when the physiological information is conditioninformation, the physiological parameter measurement device may performan operation of transmitting information to a user's terminal accordingto a condition, but the present invention is not limited thereto.

Also, for example, when the physiological information is conditioninformation, the physiological parameter measurement device may performan operation of transmitting information to a scheduling deviceaccording to a condition, but the present invention is not limitedthereto.

Also, for example, when the physiological information is conditioninformation, the physiological parameter measurement device may performan operation of outputting an advisory text for taking a rest accordingto a condition, but the present invention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of outputting an alarm according to a concentration, butthe present invention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of transmitting information to an administrator accordingto a concentration, but the present invention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of transmitting information to a user's terminal accordingto a concentration, but the present invention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of transmitting information to a scheduling deviceaccording to a concentration, but the present invention is not limitedthereto.

Also, the physiological parameter measurement device may perform anoperation corresponding to the physiological information and thephysiological parameter.

Also, the physiological parameter measurement device may perform variousoperations corresponding to the physiological information and thephysiological parameter other than the above-described examples.

FIG. 82 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 82, a physiological parameter measurement deviceoperating method 8760 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter (S8770), anoperation of acquiring physiological information (S8780), and anoperation of calculating a study or work scheduling parameter (S8790).

In this case, the above description is applicable to the operation ofacquiring the physiological parameter (S8770) and the operation ofacquiring the physiological information (S8780), and thus a redundantdescription thereof will be omitted.

Also, the operation of calculating the study or work schedulingparameter (S8790) may be performed based on the acquired physiologicalparameter and/or physiological information.

For example, the study or work scheduling parameter may be calculatedbased on an acquired heart rate and drowsiness information, but thepresent invention is not limited thereto.

Also, the operation of calculating the study or work schedulingparameter (S8790) may be performed based on the acquired physiologicalparameter or physiological information and/or study-related information.

For example, the study or work scheduling parameter may be calculatedbased on acquired drowsiness information and study time information, butthe present invention is not limited thereto.

Also, scheduling may be carried out using the study or work schedulingparameter so that study or work is performed in a time period with highefficiency.

Also, scheduling may be carried out using the study or work schedulingparameter so that a rest is taken in a time period with low efficiency.

Accordingly, scheduling may be carried out so that study or work can beperformed with the maximum efficiency in a limited time when the studyor work scheduling parameter is used.

9.5 Various Embodiments of Physiological Parameter Measurement DeviceUsed for Cognitive Rehabilitation Therapy

In the case of patients who receive cognitive rehabilitation therapy,treatment effects can be maximized only when high-intensity treatment isperformed during cognitive rehabilitation therapy.

However, supply of rehabilitation therapists to monitor individualpatients' concentration may not meet demand, so it may be difficult toincrease the individual patients' concentration to perform high-qualitytreatment.

Therefore, when a physiological parameter and physiological informationare acquired using a physiological parameter measurement device in orderto compensate for the difficulty, it is possible to monitor individualpatients' concentration, and also It is possible to perform high-qualitytreatment using the monitoring.

FIG. 83 is a diagram illustrating a physiological parameter measurementdevice used for cognitive rehabilitation therapy according to anembodiment.

Referring to FIG. 83, a physiological parameter measurement deviceaccording to an embodiment may include at least one of a firstphysiological parameter measurement device 8800, a second physiologicalparameter measurement device 8801, a third physiological parametermeasurement device 8802, a fourth physiological parameter measurementdevice 8803, and a fifth physiological parameter measurement device8804.

Also, at least one of the first to fifth physiological parametermeasurement devices 8800, 8801, 8802, 8803, and 8804 may measure aphysiological parameter of at least one of first to fourth patients8811, 8812, 8813, and 8814.

In this case, the above description is applicable to the physiologicalparameter measurement method, and thus a redundant description thereofwill be omitted.

Also, at least one of the first to fifth physiological parametermeasurement devices 8800, 8801, 8802, 8803, and 8804 may acquirephysiological information of at least one of the first to fourthpatients 8811, 8812, 8813, and 8814. For example, at least one of thefirst to fifth physiological parameter measurement devices 8800, 8801,8802, 8803, and 8804 may acquire concentration information of at leastone of the first to fourth patients 8811, 8812, 8813, and 8814.

Also, when physiological information of at least one of the first tofourth patients 8811, 8812, 8813, and 8814 is abnormal, at least one ofthe first to fifth physiological parameter measurement devices 8800,8801, 8802, 8803, and 8804 may output an alarm so that a rehabilitationtherapist 8810 can recognize the abnormality. For example., whenphysiological information of at least one of the first to fourthpatients 8811, 8812, 8813, and 8814 is abnormal, at least one of thefirst to fifth physiological parameter measurement devices 8800, 8801,8802, 8803, and 8804 may output an auditory alarm and transmitinformation related to the physiological information to a mobileterminal of the rehabilitation therapist 8810, but the present inventionis not limited thereto.

Also, when physiological information of at least one of the first tofourth patients 8811, 8812, 8813, and 8814 is abnormal, at least one ofthe first to fifth physiological parameter measurement devices 8800,8801, 8802, 8803, and 8804 may output an alarm so that the patient canrecognize the abnormality of the physiological information. For example,when physiological information of at least one of the first to fourthpatients 8811, 8812, 8813, and 8814 is abnormal, at least one of thefirst to fifth physiological parameter measurement devices 8800, 8801,8802, 8803, and 8804 may output an auditory alarm and displayinformation related to the physiological information through a displayof the corresponding patient.

Also, as described above, by monitoring a patient's physiologicalparameter and physiological information in real time and maintaining hisor her concentration during treatment, it is possible to maximize theeffect of cognitive rehabilitation therapy.

FIG. 84 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 84, a physiological parameter measurement deviceoperating method 8850 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter of a patient(S8860), an operation of acquiring physiological information of thepatient (S8870), and performing an operation corresponding to thephysiological information of the patient (S8880).

In this case, the above description is applicable to the operation ofacquiring the physiological parameter of the patient (S8860) and theoperation of acquiring the physiological information of the patient(S8870), and thus a redundant description thereof will be omitted.

Also, the operation of performing the operation corresponding to thephysiological information of the patient (S8880) may include anoperation of performing various operations corresponding to variouspieces of physiological information.

For example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of outputting an alarm according to a concentration, butthe present invention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of transmitting information to be displayed on thepatient's display according to a concentration, but the presentinvention is not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of transmitting information to a rehabilitation therapist'smobile terminal according to a concentration, but the present inventionis not limited thereto.

Also, for example, when the physiological information is concentrationinformation, the physiological parameter measurement device may performan operation of outputting an indication for increasing a concentrationaccording to the concentration, but the present invention is not limitedthereto.

Also, the physiological parameter measurement device may perform anoperation corresponding to physiological information and a physiologicalparameter.

Also, the physiological parameter measurement device may perform variousoperations corresponding to the physiological information and thephysiological parameter other than the above-described examples.

9.6 Various Embodiments of Physiological Parameter Measurement DeviceUsed in Immigration Screening

Inspection and quarantine at airports where people of various countriesand races come and go can serve as an important gatekeeper to a nationalquarantine system.

However, it may be difficult to provide in-depth management forindividuals due to a lack of management personnel compared to those whoenter airports.

Therefore, when a physiological parameter is easily measured at a placewhere individuals are subjected to immigration screening, it is possibleto more easily provide in-depth management for individuals.

FIG. 85 is a diagram illustrating a physiological parameter measurementdevice used for immigration screening according to an embodiment.

Referring to FIG. 85, physiological parameter measurement devices 8901and 8902 according to an embodiment may be placed in an immigrationkiosk 8900 and configured to measure physiological parameters ofimmigrants 8911 and 8912.

In this case, the physiological parameter measurement devices 8901 and8902 may perform face recognition or the like used for immigrationscreening, but the present invention is not limited thereto.

Also, the physiological parameter may include at least one of a heartrate, an oxygen saturation level, a blood pressure, a core temperature,or the like, but the present invention is not limited thereto.

Also, the above description is applicable to a method of thephysiological parameter measurement devices 8901 and 8902 measuring aphysiological parameter, and thus a redundant description thereof willbe omitted.

Also, the physiological parameter measurement devices 8901 and 8902 mayperform an operation corresponding to a measured physiologicalparameter. For example, as shown in FIG. 85, the first physiologicalparameter measurement device 8901 may operate to display firstinformation 8921 indicating entry allowance when a measured coretemperature of a first immigrant 8911 is in a normal range, and thesecond physiological parameter measurement device 8902 may operate todisplay second information 8922 requiring more accurate core temperaturemeasurement when a measured core temperature of a second immigrant 8912deviates from a normal range, but the present invention is not limitedthereto.

Also, as described above, by monitoring a physiological parameter of animmigrant screening during immigration to reinforce inspection andquarantine, it is possible to maximize the effect of nationalquarantine.

FIG. 86 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 86, a physiological parameter measurement deviceoperating method 8950 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter of animmigrant (S8960) and an operation of performing an operationcorresponding to the physiological parameter (S8970).

In this case, the above description is applicable to the operation ofacquiring the physiological parameter of the immigrant (S8960), and thusa redundant description thereof will be omitted.

Also, the operation of performing the operation corresponding to thephysiological parameter (S8970) may include an operation of performingvarious operations corresponding to various physiological parameters.

For example, when the physiological parameter is a heart rate, which isin a normal range, the physiological parameter measurement device mayperform a corresponding operation, but the present invention is notlimited thereto.

Also, for example, when the physiological parameter is a heart rate,which deviates from a normal range, the physiological parametermeasurement device may perform a corresponding operation, but thepresent invention is not limited thereto.

Also, for example, when the physiological parameter is an oxygensaturation level, which is in a normal range, the physiologicalparameter measurement device may perform a corresponding operation, butthe present invention is not limited thereto.

Also, for example, when the physiological parameter is an oxygensaturation level, which deviates from a normal range, the physiologicalparameter measurement device may perform a corresponding operation, butthe present invention is not limited thereto.

Also, for example, when the physiological parameter is a blood pressure,which is in a normal range, the physiological parameter measurementdevice may perform a corresponding operation, but the present inventionis not limited thereto.

Also, for example, when the physiological parameter is a blood pressure,which deviates from a normal range, the physiological parametermeasurement device may perform a corresponding operation, but thepresent invention is not limited thereto.

Also, for example, when the physiological parameter is a coretemperature, which is in a normal range, the physiological parametermeasurement device may perform a corresponding operation, but thepresent invention is not limited thereto.

Also, for example, when the physiological parameter is a coretemperature, which deviates from a normal range, the physiologicalparameter measurement device may perform a corresponding operation, butthe present invention is not limited thereto.

Also, for example, the corresponding operation when the physiologicalparameter is in a normal range may include an operation of displayingentry allowance information, but the present invention is not limitedthereto.

Also, for example, the corresponding operation when the physiologicalparameter deviates from a normal range may include an operation ofdisplaying information indicating entry denial or information indicatinga person subject to in-depth examination, but the present invention isnot limited thereto.

Also, the physiological parameter measurement device may perform variousoperations corresponding to the physiological parameter other than theabove-described examples.

9.7 Various Embodiments of Physiological Parameter Measurement DeviceUsed in Security Device

Security devices using physiological recognition such as facerecognition, iris recognition, and fingerprint recognition are beingwidely used.

However, there is also a situation in which security devices aredisabled through photos, fingerprint copying, and the like.

Accordingly, by using physiological parameters in addition tophysiological recognition, it is possible to prevent a security devicefrom being incapacitated and reinforce a security device.

FIG. 87 is a diagram illustrating a physiological parameter measurementdevice used for a security device according to an embodiment.

Referring to FIG. 87, a physiological parameter measurement device 9010according to an embodiment may be placed in a security device 9000 andconfigured to measure a physiological parameter of a subject 9020.

In detail, the security device 9000 may perform physiologicalrecognition, such as face recognition, fingerprint recognition, and irisrecognition, on the subject 9020, but the present invention is notlimited thereto.

Also, the physiological parameter measurement device 9010 may acquirethe physiological parameter of the subject 9020, but the presentinvention is not limited thereto.

In this case, the physiological parameter may include at least one of aheart rate, an oxygen saturation level, a blood pressure, a coretemperature, or the like, but the present invention is not limitedthereto.

Also, the above description is applicable to a method of thephysiological parameter measurement device 9010 measuring aphysiological parameter, and thus a redundant description thereof willbe omitted.

Also, the security device 9000 may release security on the basis ofphysiological recognition and a measured physiological parameter. Forexample, the security device 9000 may determine whether a user isdesignated through iris recognition first and then determine whether theuser is a person through the acquisition of a physiological parameter torelease security, but the present invention is not limited thereto.

Also, the security device 9000 may release security on the basis of aplurality of physiological parameters and a plurality of times ofphysiological recognition. For example, the security device 9000 maydetermine whether a user is designated through iris recognition andfingerprint recognition first and then determine whether the user is aperson through the acquisition of a heart rate and an oxygen saturationlevel to release security, but the present invention is not limitedthereto.

Also, the security device 9000 may release security on the basis of thephysiological parameter. For example, when a signal related to anacquired heartbeat matches a signal related to the designated user'sheartbeat, the security device 9000 may determine that the user is adesignated user and release security, but the present invention is notlimited thereto.

Also, as described above, by additionally utilizing the physiologicalparameter measurement device in the security device to reinforcesecurity, it is possible to maximize the effect of securityreinforcement.

FIG. 88 is a flowchart illustrating a security device operating methodaccording to an embodiment.

Referring to FIG. 88, a security device operating method 9050 accordingto an embodiment may include an operation of determining whether tomatch physiological recognition (S9060), an operation of determiningwhether to acquire a physiological parameter (S9070), and an operationof performing an operation corresponding to a result (S9080).

In this case, the physiological recognition may include at least one offingerprint recognition, face recognition, iris recognition, and thelike.

Also, the physiological parameter may include at least one of a heartrate, an oxygen saturation level, a blood pressure, a core temperature,or the like.

Also, a typical physiological recognition method is applicable to theoperation of determining whether to match physiological recognition(S9060), but the present invention is not limited thereto.

Also, the above description is applicable to a method of acquiring thephysiological parameter in order to determine whether to acquire thephysiological parameter, and thus a redundant description thereof willbe omitted.

Also, the operation of determining whether to acquire the physiologicalparameter (S9070) may include an operation of determining whether toacquire various physiological parameters. For example, the operation ofdetermining whether to acquire the physiological parameter (S9070) mayinclude an operation of determining whether a heart rate is to beacquired, but the present invention is not limited thereto.

Also, the operation of determining whether to acquire the physiologicalparameter (S9070) may include an operation of determining whether tomatch a physiological signal for acquiring the physiological parameter.For example, the operation of determining whether to acquire thephysiological parameter (S9070) may include an operation of determiningwhether a heartbeat signal for acquiring a heart rate matches aheartbeat signal of the designated user, but the present invention isnot limited thereto.

Also, the operation of determining whether to acquire the physiologicalparameter (S9070) may include an operation of determining whether tomatch the physiological parameter. For example, the operation ofdetermining whether to acquire the physiological parameter (S9070) mayinclude an operation of determining whether a currently measured heartrate is included in a pre-measured heart rate of the subject and areference range based on the pre-measured heart rate, but the presentinvention is not limited thereto.

Also, the operation of performing the operation corresponding to theresult (S9080) may include an operation of releasing security when allsecurity release conditions are satisfied and an operation ofmaintaining security when all security release conditions are notsatisfied, but the present invention is not limited thereto.

Also, the security device may perform various operations for reinforcingsecurity using a physiological parameter other than the above-describedexamples.

9.8 Various Embodiments of Physiological Parameter Measurement DeviceUsed in Kiosk

When a physiological parameter measurement device capable ofconveniently measuring a physiological parameter is placed in a kioskfor displaying information, it is possible to continuously monitorindividual health by conveniently monitoring physiological parameters ineveryday life.

FIG. 89 is a diagram illustrating a physiological parameter measurementdevice used for a kiosk according to an embodiment.

Referring to FIG. 89, a physiological parameter measurement device 9110according to an embodiment may be placed in a kiosk 9100 and configuredto measure a physiological parameter of a subject 9120.

In this case, the kiosk 9100 may include a display for displaying atleast one piece of information. For example, as shown in FIG. 89, thekiosk 9100 may display date information and weather information, but thepresent invention is not limited thereto.

Also, the physiological parameter measurement device 9110 may acquire atleast one physiological parameter. For example, the physiologicalparameter measurement device 9110 may acquire a heart rate. However, thepresent invention is not limited thereto, and the physiologicalparameter measurement device 9110 may acquire at least one physiologicalparameter among a heart rate, an oxygen saturation level, a bloodpressure, and a core temperature.

Also, the physiological parameter measurement device 9110 may acquire atleast one piece of physiological information. For example, thephysiological parameter measurement device 9110 may acquire today'scondition information. However, the present invention is not limitedthereto, and the physiological parameter measurement device 9110 mayacquire at least one piece of physiological information among drowsinessinformation, condition information, concentration information, healthinformation, etc.

Also, the kiosk 9100 may display the acquired physiological parameter.For example, as shown in FIG. 89, the kiosk 9100 may display theacquired heart rate, oxygen saturation level, and blood pressure, butthe present invention is not limited thereto.

Also, the kiosk 9100 may display the acquired physiological information.For example, as shown in FIG. 89, the kiosk 9100 may display theacquired today's condition information, but the present invention is notlimited thereto.

Also, the kiosk 9100 may transmit the acquired physiological parameterand physiological information. For example, the kiosk 9100 may transmitthe acquired physiological parameter and physiological information to amobile terminal of the subject 9120, but the present invention is notlimited thereto.

Also, the kiosk 9100 may transmit the acquired physiological parameterand physiological information. For example, the kiosk 9100 may transmitthe acquired physiological parameter and physiological information to aterminal through which an administrator can check the physiologicalparameter and physiological information, but the present invention isnot limited thereto.

Also, in this case, a patient with an abnormal physiological parametermay be detected in a building where the kiosk 9100 is placed, and thusit is possible to reinforce the quarantine and security of the building.

Also, by additionally using a physiological parameter measurement devicein a kiosk as described above, individual health may be continuouslymonitored, and thus it is possible to reinforce the quarantine andsecurity of a building where the kiosk is placed.

Also, the above-described physiological parameter measurement device maybe used for human resource management. For example, the physiologicalparameter measurement device may be used in a kiosk for access ofconstruction site workers to cope with construction site workers withabnormal physiological parameters, but the present invention is notlimited thereto.

FIG. 90 is a flowchart illustrating a physiological parametermeasurement device operating method according to an embodiment.

Referring to FIG. 90, a physiological parameter measurement deviceoperating method 9150 according to an embodiment may include at leastsome of an operation of acquiring a physiological parameter (S9160), anoperation of acquiring physiological information (S9170), and outputtingthe physiological parameter and the physiological information (S9180).

In this case, the above-described physiological parameter acquisitionmethod is applicable to the operation of acquiring the physiologicalparameter (S9160), and thus a redundant description thereof will beomitted.

Also, the above-described physiological information acquisition methodis applicable to the operation of acquiring the physiologicalinformation (S9170), and thus a redundant description thereof will beomitted.

Also, the operation of outputting the physiological parameter and thephysiological information (S9180) may include an operation of displayingthe physiological parameter and the physiological information. Forexample, the operation of outputting the physiological parameter and thephysiological information (S9180) may include an operation of displayinga heart rate, an oxygen saturation level, a blood pressure, and today'scondition information, but the present invention is not limited thereto.

Also, the operation of outputting the physiological parameter and thephysiological information (S9180) may include an operation oftransmitting the physiological parameter and the physiologicalinformation. For example, the operation of outputting the physiologicalparameter and the physiological information (S9180) may include anoperation of transmitting a heart rate, an oxygen saturation level, ablood pressure, and today's condition information to another mobileterminal, but the present invention is not limited thereto.

Also, the operation of outputting the physiological parameter and thephysiological information (S9180) may include an operation of printingthe physiological parameter and the physiological information. Forexample, the operation of outputting the physiological parameter and thephysiological information (S9180) may include an operation of printing aheart rate, an oxygen saturation level, a blood pressure, and today'scondition information, but the present invention is not limited thereto.

Also, the physiological parameter acquisition device may perform variousoperations of acquiring and outputting physiological parameters andphysiological information other than the above-described examples.

According to an embodiment of the present invention, it is possible toprovide a method of acquiring a physiological parameter in a contactlessmanner.

According to another embodiment of the present invention, it is possibleto provide a method of reducing noise caused by subject movement.

According to still another embodiment of the present invention, it ispossible to provide a method of reducing noise caused by a change inintensity of external light.

According to still another embodiment of the present invention, it ispossible to provide a method of acquiring various physiologicalparameters at the same time.

According to still another embodiment of the present invention, it ispossible to provide a method of acquiring physiological information onthe basis of various physiological parameters.

According to still another embodiment of the present invention, it ispossible to provide a method of acquiring various physiologicalparameters in association with each other at the same time.

According to still another embodiment of the present invention, it ispossible to provide a smart mirror device configured to acquire at leasttwo associated physiological parameters.

According to still another embodiment of the present invention, it ispossible to provide a smart mirror device operating method to acquire atleast two associated physiological parameters.

According to still another embodiment of the present invention, it ispossible to provide a method and device for detecting drowsiness on thebasis of an LF/HF ratio of a heartbeat signal and a heart rate of asubject.

According to still another embodiment of the present invention, it ispossible to provide a smart mirror device including a switch device.

The method according to an embodiment may be implemented in the form ofprogram instructions executable by a variety of computer means and maybe recorded on a computer-readable medium. The computer-readable mediummay include program instructions, data files, data structures, and thelike alone or in combination. The program instructions recorded on themedium may be designed and configured specifically for an embodiment ormay be publicly known and usable by those who are skilled in the fieldof computer software. Examples of the computer-readable recording mediuminclude a magnetic medium, such as a hard disk, a floppy disk, and amagnetic tape, an optical medium, such as a compact disc read-onlymemory (CD-ROM), a digital versatile disc (DVD), etc., a magneto-opticalmedium such as a floptical disk, and a hardware device speciallyconfigured to store and perform program instructions, for example, aread-only memory (ROM), a random access memory (RAM), a flash memory,etc. Examples of the computer instructions include not only machinelanguage code generated by a compiler, but also high-level language codeexecutable by a computer using an interpreter or the like. The hardwaredevice may be configured to operate as one or more software modules inorder to perform the operations of an embodiment, and vice versa.

Although the present invention has been described with reference tospecific embodiments and drawings, it will be appreciated that variousmodifications arid changes can be made from the disclosure by thoseskilled in the art. For example, appropriate results may be achievedalthough the described techniques are performed in an order differentfrom that described above and/or although the described components suchas a system, a structure, a device, or a circuit are combined in amanner different from that described above and/or replaced orsupplemented by other components or their equivalents.

Therefore, other implementations, embodiments, and equivalents arewithin the scope of the following claims.

What is claimed is:
 1. A method for measuring a physiological parameterof a subject in a contactless manner using a camera, the method executedon one or more processors, the method comprising: receiving a pluralityof image frames for the subject, wherein the plurality of image framesare captured by the camera; obtaining a first color channel value, asecond color channel value, a third color channel value for at least oneof the plurality of image frames, wherein the first color channel valuerepresents average pixel value of a first color channel for at least aportion of at least one of the plurality of image frames, the secondcolor channel value represents average pixel value of a second colorchannel for at least a portion of at least one of the plurality of imageframes, and wherein the third. channel value represents average pixelvalue of a third color channel for at least a portion of at least one ofthe plurality of image frames; calculating a first difference values anda second difference values based on the first color channel value, thesecond color channel value and the third color channel value for atleast one of the plurality of image frames, wherein each of the firstdifference values represents a difference value of the first colorchannel value and the second color channel value for the same imageframe, and each of the second difference values represents a differencevalue of the first color channel value and the third color channel valuefor the same image frame; obtaining a first characteristic values for afirst group of image frames captured in a first preset time intervalbased on at least one of the first difference values for the first groupof image frames and an average of the first difference values for thefirst group of image frames; obtaining a second characteristic valuesfor the first group of image frames based on at least one of the seconddifference values for the first group of image frames and an average ofthe second difference values for the first group of image frames; anddetermining the physiological parameter of the subject based on thefirst characteristic value and the second characteristic value.
 2. Themethod of the claim 1, wherein the physiological parameter includes atleast one of heart rate and blood pressure.
 3. The method of the claim1, wherein the first color channel, the second color channel and thethird color channel are color channels of RGB color space.
 4. The methodof the claim 3, wherein the first color channel is set to a greenchannel, the second color channel is set to a red channel and the thirdcolor channel is set to a blue channel in order to reduce noise inconsideration of the absorbance of hemoglobin and oxyhemoglobin.
 5. Themethod of the claim 1, wherein the first characteristic values areobtained based on a first deviation values of the first differencevalues for at least one of the image frames included in the first groupof image frames, wherein the second characteristic values are obtainedbased on a second deviation values of the second difference values forat least one of the image frames included in the first group of imageframes, wherein the first deviation values are calculated based on thefirst difference values for at least one of image frames included in thefirst group of image frames and an average value of the first differencevalues for the first group of image frames, wherein the second deviationvalues are calculated based on the second difference values for at leastone of image frames included in the first group of image frames and anaverage value of the second difference values for the first group ofimage frames.
 6. The method of the claim 5, wherein the firstcharacteristic values and the second characteristic values arenormalized values.
 7. The method of the claim 6, wherein the firstcharacteristic values are values normalized by a first standarddeviation value, and the second characteristic values are valuesnormalized by a second standard deviation value, wherein the firststandard deviation value is a standard deviation value of the firstdifference values for the first group of image frames, and the secondstandard deviation value is a standard deviation value of the seconddifference values for the first group of image frames.
 8. The method ofthe claim 1, wherein the physiological parameter of the subject isdetermined based on a third characteristic values obtained as a sum ofthe first characteristic values and the second characteristic values. 9.The method of the claim 1, wherein the method further comprising:outputting the physiological parameter of the subject, wherein thedetermined physiological parameter includes a first physiologicalparameter and a second physiological parameter, wherein the outputtedphysiological parameter is determined based on the first physiologicalparameter and the second physiological parameter.
 10. The method of theclaim 9, wherein the first physiological parameter is determined basedon a second group of image frames and the second physiological parameteris determined based on a third group of image frames, wherein the numberof the image frames included in the first group of image frames issmaller than the number of image frames included in the second and thethird group of image frames, wherein the first group of image frames isincluded in the second group of image frames.
 11. The method of theclaim 10, wherein the number of the image frames included in the secondgroup of image frames is same as the number of the image frames includedin the third group of image frames.
 12. The method of the claim 1,wherein the method further comprising: outputting the physiologicalparameter of the subject based on the determined physiologicalparameter, wherein the determined physiological parameter includes atleast four of a pre-physiological parameter values, wherein theoutputted physiological parameter is determined based on the four of thepre-physiological parameter values.
 13. The method of claim 1, whereinthe method further comprising: outputting the physiological parameter ofthe subject based on the determined physiological parameter, wherein theoutputted physiological parameter includes a first physiologicalparameter and a second physiological parameter, wherein the secondphysiological parameter is a physiological parameter of the same type asthe first physiological parameter, and the second physiologicalparameter is outputted after the first physiological parameter isoutputted, when a difference between the second physiological parameterand the first physiological parameter exceeds a reference value, thesecond physiological parameter is amended and outputted.
 14. A methodfor measuring a physiological parameter of a subject in a contactlessmanner using an infrared camera, the method executed on one or moreprocessors, the method comprising: receiving a plurality of image framesfor the subject, wherein the plurality of image frames are captured bythe infrared camera; obtaining a first region component value, a secondregion component value, a third region component value for at least oneof the plurality of image frames, wherein the first region componentvalue represents an average pixel value of a first region of interestfor at least one of the plurality of image frames, the second regioncomponent value represents an average pixel value of a second region ofinterest for at least one of the plurality of image frames and the thirdregion component value represents an average pixel value of a thirdregion of interest for at least one of the plurality of image frames;calculating a first difference values and a second difference valuesbased on the first region component value, the second region componentvalue and the third region component value for at least one of theplurality of image frames, wherein each of the first difference valuesrepresents a difference value of the first region component value andthe second region component value for the same image frame and each ofthe second difference values represents a difference value of the firstregion component value and the third region component value for the sameimage frame; obtaining a first characteristic values for a first groupof image frames captured in a first preset time interval based on atleast one of the first difference values for the first group of imageframes and an average of the first difference values for the first groupof image frames; obtaining a second characteristic values for the firstgroup of image frames based on at least one of the second differencevalues for the first group of image frames and an average of the seconddifference values for the first group of image frames; and determiningthe physiological parameter of the subject based on the firstcharacteristic value and the second characteristic value.
 15. The methodof the claim 14, wherein the physiological parameter includes at leastone of heart rate and blood pressure.
 16. The method of the claim 14,wherein the first characteristic values are obtained based on a firstdeviation values of the first difference values for at least one of theimage frames included in the first group of image frames, wherein thesecond characteristic values are obtained based on a second deviationvalues of the second difference values for at least one of the imageframes included in the first group of image frames, wherein the firstdeviation values is calculated based on the first difference values forat least one of image frames included in the first group of image framesand an average value of the first difference values for the first groupof image frames, wherein the second deviation values is calculated basedon the second difference values for at least one of image framesincluded in the first group of image frames and an average value of thesecond difference values for the first group of image frames.
 17. Anon-transitory recording medium having a program recorded thereon forexecuting the method of claim
 1. 18. A non-transitory recording mediumhaving a program recorded thereon for executing the method of claim 14.19. A physiological parameter measuring device for measuring aphysiological parameter of a subject in a contactless manner,comprising: an image obtaining unit for obtaining plurality of imageframes for the subject; a control unit configured to obtain aphysiological parameter of the subject using the plurality of imageframes; wherein the control unit is configured to: obtain a first colorchannel value, a second color channel value, a third color channel valuefor at least one of the plurality of image frames, wherein the firstcolor channel value represents average pixel value of a first colorchannel for at least one of the plurality of image frames, the secondcolor channel value represents average pixel value of a second colorchannel for at least one of the plurality of image frames, and whereinthe third channel value represents average pixel value of a third colorchannel for at least one of the plurality of image frames; calculate afirst difference values and a second difference values based on thefirst color channel value, the second color channel value and the thirdcolor channel value for at least one of the plurality of image frames,wherein each of the first difference values represents a differencevalue of the first color channel value and the second color channelvalue for the same image frame, and each of the second difference valuesrepresents a difference value of the first color channel value and thethird color channel value for the same image frame; obtain a firstcharacteristic values for a first group of image frames captured in afirst preset time interval based on at least one of the first differencevalues for the first group of image frames and an average of the firstdifference values for the first group of image frames; obtain a secondcharacteristic values for the first group of image frames based on atleast one of the second difference values for the first group of imageframes and an average of the second difference values for the firstgroup of image frames; and determine the physiological parameter of thesubject based on the first characteristic value and the secondcharacteristic value.
 20. The device of the claim 19, wherein thecontrol unit is configured to obtain a physiological information basedon the determined physiological parameter, wherein the physiologicalinformation includes at least one of emotion information, drowsinessinformation, stress information, and excitability information.